Skip to content

Home

aws-lambda-function-to-email-notification-for-new-file-uploaded

import json
import urllib.parse
import boto3

import os
sns_topic_arn = os.environ['sns_topic_arn']

print('Loading function')

s3_client  = boto3.client('s3')
sns_client = boto3.client('sns')

# useful links
# https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_url
# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html
# https://docs.aws.amazon.com/lambda/latest/dg/with-s3-tutorial.html


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    #print('This is sns_topic_arn: {}.'.format(sns_topic_arn))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')

    try:
        response = s3_client.get_object(Bucket=bucket, Key=key)
        #print("CONTENT TYPE: " + response['ContentType'])
        #print('Object name `{}` from bucket `{}`.'.format(key, bucket))

        url = s3_client.generate_presigned_url(
            ClientMethod='get_object',
            Params={'Bucket': bucket, 'Key': key},
            ExpiresIn='25200' #7 days
        )

        #print('This is the result URL - {} '.format(url))

        #presigned_url_for_email = '<a href="{}">Your private link for download</a>'.format(url)
        #print(presigned_url_for_email)

        response = sns_client.publish(
             TopicArn=sns_topic_arn,
             Message='Notification for bucket {}. \nA new file name {} was added. \n\nHere is the link for download: \n\n {} \n\nPlease let us know if you need a new link as for security they expire after 1 hour. \nThank you. \nThe fantastic staff.'.format(bucket, key, url),
             Subject='Bucket notification - new file {}'.format(key)
        )

        return 0
        #response['ContentType']

    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        raise e

Sample lambda functions that run based on S3 event trigger on s3:ObjectCreated.

UPDATE

Upload objects greater than 16MB+ requires the additional event trigger of s3:ObjectCreated:CompleteMultipartUpload.

Kudus for AWS Support for helping me with the troubleshoot.

1) SNS topic with email, txt, ... subscribers.

2) Create S3 event trigger with the lambda funtion as a target

s3-object-created-trigger-lambda-and-sns-notification

S3 events for:

  • s3:ObjectCreated:Put

  • s3:ObjectCreated:CompleteMultipartUpload

3) Target a lambda functions to run below logic

lambda sample code for s3 presign url

  • DISCLAIMER >> Use at your own responsability. <<

Sample Lambda code


UPDATE and other details

Upload objects greater than 16MB+ requires the additional event trigger of s3:ObjectCreated:CompleteMultipartUpload.

Kudus for AWS Support for helping me with the troubleshoot.

S3 events trigger for:

  • s3:ObjectCreated:Put

  • s3:ObjectCreated:CompleteMultipartUpload

Objects greater than 16MB are getting uploaded as a Multipart uploads.

Multipart upload allows us to upload a single object as a set of parts. Each part is a contiguous portion of the object's data.

S3 bucket event s3:ObjectCreated:Put provides notification when an object is created by an HTTP PUT operation.

S3 bucket event s3:ObjectCreated:CompleteMultipartUpload provides notification for an object which was created by the completion of a S3 multi-part upload.

Documentation

Happy learning

Antonio Feijao

Paintball 25 event

2022 Paintball 25

  • Private event

  • Event location address

Delta Force Paintball East London 1 Aveley Rd, Upminster RM14 2TN

Google maps link - https://goo.gl/maps/jcaCaLXtZ1aVEGMv8

Message from the organisation:

"Many thanks for your booking.

Please note that all players must complete an Online Registration Form before they are allowed to play.

This should be done before arriving at the centre so as to ensure smooth entry on the day of your event.

Completing registration in advance allows you to start playing sooner, and mobile reception may be limited at the centre and therefore your registration process may be difficult to complete on the day.

To complete your Online Registration Form please click the button below.

    (Antonio to share the link in private)

This email can be forwarded on to your other players, or you can forward the link to your unique Online Registration Portal to your group via SMS, Facebook or WhatsApp by copying the link below.


Once you have completed your Online Registration Form you can click the Home button in the top left corner of the registration portal and then click the yellow Registered button in the top right corner to see a list of all of the registered players. If there are any players yet to register then please remind them in advance so as to fast track your arrival on the day.


If you have any further queries do not hesitate to give me a call or email me.

Kind regards,
Nancy Mills
Event Co-ordinator
0203 869 9135*

Rubik's cube solved in 3 minutes by Antonio Feijao

TIP - Use the video controls to change the playback speed.


I learned these steps initially from this video https://youtu.be/7Ron6MN45LY - Learn How to Solve a Rubik's Cube in 10 Minutes (Beginner Tutorial)

You too can learn how to solve the rubik cube

(Detailed videos for each step coming soon...)


01-all-white-middles-to-the-top


02-turn-whites-middles-to-white-center


03-yellow-up-fix-white-corners


04-yellow-up-fix-the-middle-colors


05-do-yellow-cross


06-put-the-yellow-middles-in-the-right-place


07-put-the-yellow-corners-in-the-right-place


08-rotate-the-yellow-corner-to-fix-position


09-you-have-solved-the-rubik-cube


Happy learning,

Antonio Feijao

AWS Security Hub work in Python 3 and Boto3

Testing some commands with python3 and boto3

List of AWS Services from Boto3 clients

The below error message, shows very usefull information. I can see all the boto3.client's name that I can use.

Does anyone knows how to get the list without using the error?

Input

import boto3

boto3.client(dir)

output (tweaked to show the services names in a multi line list)

UnknownServiceError: Unknown service: '<built-in function dir>'.

Valid service names are:

accessanalyzer,
account,
acm,
acm-pca,
alexaforbusiness,
amp,
amplify,
amplifybackend,
amplifyuibuilder,
apigateway,
apigatewaymanagementapi,
apigatewayv2,
appconfig,
appconfigdata,
appflow,
appintegrations,
application-autoscaling,
application-insights,
applicationcostprofiler,
appmesh,
apprunner,
appstream,
appsync,
athena,
auditmanager,
autoscaling,
autoscaling-plans,
backup,
backup-gateway,
batch,
braket,
budgets,
ce,
chime,
chime-sdk-identity,
chime-sdk-meetings,
chime-sdk-messaging,
cloud9,
cloudcontrol,
clouddirectory,
cloudformation,
cloudfront,
cloudhsm,
cloudhsmv2,
cloudsearch,
cloudsearchdomain,
cloudtrail,
cloudwatch,
codeartifact,
codebuild,
codecommit,
codedeploy,
codeguru-reviewer,
codeguruprofiler,
codepipeline,
codestar,
codestar-connections,
codestar-notifications,
cognito-identity,
cognito-idp,
cognito-sync,
comprehend,
comprehendmedical,
compute-optimizer,
config,
connect,
connect-contact-lens,
connectparticipant,
cur,
customer-profiles,
databrew,
dataexchange,
datapipeline,
datasync,
dax,
detective,
devicefarm,
devops-guru,
directconnect,
discovery,
dlm,
dms,
docdb,
drs,
ds,
dynamodb,
dynamodbstreams,
ebs,
ec2,
ec2-instance-connect,
ecr,
ecr-public,
ecs,
efs,
eks,
elastic-inference,
elasticache,
elasticbeanstalk,
elastictranscoder,
elb,
elbv2,
emr,
emr-containers,
es,
events,
evidently,
finspace,
finspace-data,
firehose,
fis,
fms,
forecast,
forecastquery,
frauddetector,
fsx,
gamelift,
glacier,
globalaccelerator,
glue,
grafana,
greengrass,
greengrassv2,
groundstation,
guardduty,
health,
healthlake,
honeycode,
iam,
identitystore,
imagebuilder,
importexport,
inspector,
inspector2,
iot,
iot-data,
iot-jobs-data,
iot1click-devices,
iot1click-projects,
iotanalytics,
iotdeviceadvisor,
iotevents,
iotevents-data,
iotfleethub,
iotsecuretunneling,
iotsitewise,
iotthingsgraph,
iottwinmaker,
iotwireless,
ivs,
kafka,
kafkaconnect,
kendra,
kinesis,
kinesis-video-archived-media,
kinesis-video-media,
kinesis-video-signaling,
kinesisanalytics,
kinesisanalyticsv2,
kinesisvideo,
kms,
lakeformation,
lambda,
lex-models,
lex-runtime,
lexv2-models,
lexv2-runtime,
license-manager,
lightsail,
location,
logs,
lookoutequipment,
lookoutmetrics,
lookoutvision,
machinelearning,
macie,
macie2,
managedblockchain,
marketplace-catalog,
marketplace-entitlement,
marketplacecommerceanalytics,
mediaconnect,
mediaconvert,
medialive,
mediapackage,
mediapackage-vod,
mediastore,
mediastore-data,
mediatailor,
memorydb,
meteringmarketplace,
mgh,
mgn,
migration-hub-refactor-spaces,
migrationhub-config,
migrationhubstrategy,
mobile,
mq,
mturk,
mwaa,
neptune,
network-firewall,
networkmanager,
nimble,
opensearch,
opsworks,
opsworkscm,
organizations,
outposts,
panorama,
personalize,
personalize-events,
personalize-runtime,
pi,
pinpoint,
pinpoint-email,
pinpoint-sms-voice,
polly,
pricing,
proton,
qldb,
qldb-session,
quicksight,
ram,
rbin,
rds,
rds-data,
redshift,
redshift-data,
rekognition,
resiliencehub,
resource-groups,
resourcegroupstaggingapi,
robomaker,
route53,
route53-recovery-cluster,
route53-recovery-control-config,
route53-recovery-readiness,
route53domains,
route53resolver,
rum,
s3,
s3control,
s3outposts,
sagemaker,
sagemaker-a2i-runtime,
sagemaker-edge,
sagemaker-featurestore-runtime,
sagemaker-runtime,
savingsplans,
schemas,
sdb,
secretsmanager,
securityhub,
serverlessrepo,
service-quotas,
servicecatalog,
servicecatalog-appregistry,
servicediscovery,
ses,
sesv2,
shield,
signer,
sms,
sms-voice,
snow-device-management,
snowball,
sns,
sqs,
ssm,
ssm-contacts,
ssm-incidents,
sso,
sso-admin,
sso-oidc,
stepfunctions,
storagegateway,
sts,
support,
swf,
synthetics,
textract,
timestream-query,
timestream-write,
transcribe,
transfer,
translate,
voice-id,
waf,
waf-regional,
wafv2,
wellarchitected,
wisdom,
workdocs,
worklink,
workmail,
workmailmessageflow,
workspaces,
workspaces-web,
xray

A curiosity...
I echod all the aww services' names to wc -l to get a count of the services.

See for yourself how many boto3.client client has,
which, should give us an idea of how many services AWS has.

echo "accessanalyzer, account, acm, acm-pca, alexaforbusiness, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appflow, appintegrations, application-autoscaling, application-insights, applicationcostprofiler, appmesh, apprunner, appstream, appsync, athena, auditmanager, autoscaling, autoscaling-plans, backup, backup-gateway, batch, braket, budgets, ce, chime, chime-sdk-identity, chime-sdk-meetings, chime-sdk-messaging, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codeartifact, codebuild, codecommit, codedeploy, codeguru-reviewer, codeguruprofiler, codepipeline, codestar, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, config, connect, connect-contact-lens, connectparticipant, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, dax, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, drs, ds, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, elastic-inference, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, emr-containers, es, events, evidently, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, fsx, gamelift, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, honeycode, iam, identitystore, imagebuilder, importexport, inspector, inspector2, iot, iot-data, iot-jobs-data, iot1click-devices, iot1click-projects, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleethub, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, kafka, kafkaconnect, kendra, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, lightsail, location, logs, lookoutequipment, lookoutmetrics, lookoutvision, machinelearning, macie, macie2, managedblockchain, marketplace-catalog, marketplace-entitlement, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediastore, mediastore-data, mediatailor, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhubstrategy, mobile, mq, mturk, mwaa, neptune, network-firewall, networkmanager, nimble, opensearch, opsworks, opsworkscm, organizations, outposts, panorama, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, polly, pricing, proton, qldb, qldb-session, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, rekognition, resiliencehub, resource-groups, resourcegroupstaggingapi, robomaker, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53resolver, rum, s3, s3control, s3outposts, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-runtime, savingsplans, schemas, sdb, secretsmanager, securityhub, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, sms, sms-voice, snow-device-management, snowball, sns, sqs, ssm, ssm-contacts, ssm-incidents, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, support, swf, synthetics, textract, timestream-query, timestream-write, transcribe, transfer, translate, voice-id, waf, waf-regional, wafv2, wellarchitected, wisdom, workdocs, worklink, workmail, workmailmessageflow, workspaces, workspaces-web, xray" | tr ' ' '\n' | wc -l
299
(...)  | tr ' ' '\n' | wc -l
299

299 AWS boto3 client or 299 AWS Services? (checked on 2022-03-24)


Working with aws boto3 client securityhub

Here is an example of the second service name. The number [1] shows a second position in the array. [0] if the first position, first value.

input

import boto3

securityhub = boto3.client('securityhub')

securityhub.describe_products()['Products'][1]

output

{'ProductArn': 'arn:aws:securityhub:xxxxxx:xxxxxx:product/armordefense/armoranywhere',
 'ProductName': 'Armor Anywhere',
 'CompanyName': 'ARMOR',
 'Description': 'Armor Anywhere delivers managed security and compliance for AWS.',
 'Categories': ['Managed Security Service Provider (MSSP)'],
 'IntegrationTypes': ['SEND_FINDINGS_TO_SECURITY_HUB'],
 'MarketplaceUrl': 'https://aws.amazon.com/marketplace/seller-profile?id=797425f4-6823-xxxxxx',
 'ActivationUrl': 'https://amp.armor.com/account/cloud-connections',
 'ProductSubscriptionResourcePolicy': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"xxxxxx"},"Action":["securityhub:BatchImportFindings"],"Resource":"arn:aws:securityhub:xxxxxx:xxxxxx:product-subscription/armordefense/armoranywhere","Condition":{"StringEquals":{"securityhub:TargetAccount":"xxxxxx"}}},{"Effect":"Allow","Principal":{"AWS":"xxxxxx"},"Action":["securityhub:BatchImportFindings"],"Resource":"arn:aws:securityhub:xxxxxx:xxxxxx:product/armordefense/armoranywhere","Condition":{"StringEquals":{"securityhub:TargetAccount":"xxxxxx"}}}]}'}

Now, let's use a for loop to list through all services names.

initialising the variable companies

and running a for loop to add all companies into the variable companies

companies = []

for CompanyName in securityhub.describe_products()['Products']:
    print(CompanyName['CompanyName'])
    companies.append(CompanyName['CompanyName'])

(...)

Then, to remove duplicate services names in the variable companies, we can use dict.fromkeys to remove duplicates.

```python list(dict.fromkeys(companies))

['3CORESec', 'ARMOR', 'AWS', 'Alert Logic', 'Amazon', 'Aqua Security', 'Atlassian', 'AttackIQ', 'Barracuda Networks', 'BigID', 'Blue Hexagon', 'Capitis', 'Caveonix', 'Check Point', 'Cloud Custodian', 'Cloud Storage Security', 'CrowdStrike', 'CyberArk', 'DisruptOps, Inc.', 'FireEye', 'Forcepoint', 'Fugue', 'Guardicore', 'HackerOne', 'IBM'] ```

Finding duplicated files using command line CLI in Linux MacOS Ubuntu

Easy way to find duplicated files in a folder or in all disk

find . -type f -name "*" -print0 | xargs -0 -I {} shasum -a 256 {}

Or course, output the above command to a file, cut -f 1, then sort and pipe it into uniq -c to count duplicates

find . -type f -name "*.JPG" -print0 | xargs -0 -I {} shasum -a 256 {} > finding-duplicates.txt

cat finding-duplicates.txt | cut -f 1 -d ' ' | sort | uniq -c | sort -rn | head -n10

grep 30848de6dba6f90bef4027fbf97a66dcc6f1f2eb3e8a6e47f0e9ce3fc411ce79 finding-duplicates.txt

of course, we now can automate this to keep the first file but move the duplicated into a backup folder before deleting them.

Example of my output on a folder with old photos that got duplicated over type... some photo is now 6x duplicated... time to automate tidying up!

Happy learning,

Antonio Feijao

Python3 start learning first commands

So, if you just start learning Python 3, where are some tips to help you start.

Assume you already installed Python 3 and is worked properly, here is simple commands to start.


import this in Python 3

Start the python3 then type import this, and you should get the output below.

Basically is a "best practices" for coding.

import this 
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

ask for help in Python 3

The help() command will be like a manual and samples commands

help()

Welcome to Python 3.8's help utility!

If this is your first time using Python, you should definitely check out
the tutorial on the Internet at https://docs.python.org/3.8/tutorial/.

Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules.  To quit this help utility and
return to the interpreter, just type "quit".

To get a list of available modules, keywords, symbols, or topics, type
"modules", "keywords", "symbols", or "topics".  Each module also comes
with a one-line summary of what it does; to list the modules whose name
or summary contain a given string such as "spam", type "modules spam".

List available modules in Python 3 with help command

After executing help(), simply type modules.

help> modules

Please wait a moment while I gather a list of all available modules...

IPython             _tracemalloc        gzip                secrets
__future__          _uuid               hashlib             select
_abc                _warnings           heapq               selectors
_ast                _weakref            hmac                shelve
_asyncio            _weakrefset         html                shlex
_bisect             _xxsubinterpreters  http                shutil
_blake2             _xxtestfuzz         imaplib             signal
_bootlocale         abc                 imghdr              site
_bz2                aifc                imp                 sitecustomize
_codecs             antigravity         importlib           six
_codecs_cn          argparse            inspect             smtpd
_codecs_hk          array               io                  smtplib
_codecs_iso2022     ast                 ipaddress           sndhdr
_codecs_jp          asynchat            ipython_genutils    socket
_codecs_kr          asyncio             itertools           socketserver
_codecs_tw          asyncore            jedi                spwd
_collections        atexit              json                sqlite3
_collections_abc    audioop             keyword             sre_compile
_compat_pickle      autoreload          linecache           sre_constants
_compression        backcall            locale              sre_parse
_contextvars        base64              logging             ssl
_crypt              bdb                 lzma                stat
_csv                binascii            mailbox             statistics
_ctypes             binhex              mailcap             storemagic
_ctypes_test        bisect              marshal             string
_curses             builtins            math                stringprep
_curses_panel       bz2                 mimetypes           struct
_datetime           cProfile            mmap                subprocess
_dbm                calendar            modulefinder        sunau
_decimal            cgi                 multiprocessing     symbol
_dummy_thread       cgitb               netrc               sympyprinting
_elementtree        chunk               nis                 symtable
_functools          cmath               nntplib             sys
_hashlib            cmd                 ntpath              sysconfig
_heapq              code                nturl2path          syslog
_imp                codecs              numbers             tabnanny
_io                 codeop              opcode              tarfile
_json               collections         operator            telnetlib
_locale             colorsys            optparse            tempfile
_lsprof             compileall          os                  termios
_lzma               concurrent          ossaudiodev         test
_markupbase         configparser        parser              tests
_md5                contextlib          parso               textwrap
_multibytecodec     contextvars         pathlib             this
_multiprocessing    copy                pdb                 threading
_opcode             copyreg             pexpect             time
_operator           crypt               pickle              timeit
_osx_support        csv                 pickleshare         token
_pickle             ctypes              pickletools         tokenize
_posixshmem         curses              pipes               trace
_posixsubprocess    cythonmagic         pkg_resources       traceback
_py_abc             dataclasses         pkgutil             tracemalloc
_pydecimal          datetime            platform            traitlets
_pyio               dbm                 plistlib            tty
_queue              decimal             poplib              turtle
_random             decorator           posix               types
_sha1               difflib             posixpath           typing
_sha256             dis                 pprint              unicodedata
_sha3               distutils           profile             unittest
_sha512             doctest             prompt_toolkit      urllib
_signal             dummy_threading     pstats              uu
_sitebuiltins       email               pty                 uuid
_socket             encodings           ptyprocess          venv
_sqlite3            enum                pwd                 warnings
_sre                errno               py_compile          wave
_ssl                faulthandler        pyclbr              wcwidth
_stat               fcntl               pydoc               weakref
_statistics         filecmp             pydoc_data          webbrowser
_string             fileinput           pyexpat             wsgiref
_strptime           fnmatch             pygments            xdrlib
_struct             formatter           queue               xml
_symtable           fractions           quopri              xmlrpc
_sysconfigdata__linux_x86_64-linux-gnu ftplib              random              xxlimited
_sysconfigdata__x86_64-linux-gnu functools           re                  xxsubtype
_testbuffer         gc                  readline            zipapp
_testcapi           genericpath         reprlib             zipfile
_testimportmultiple getopt              resource            zipimport
_testinternalcapi   getpass             rlcompleter         zlib
_testmultiphase     gettext             rmagic              
_thread             glob                runpy               
_threading_local    grp                 sched               

Enter any module name to get more help.  Or, type "modules spam" to search
for modules whose name or summary contain the string "spam".

help> 

List available TOPICS in Python 3 with help command

help> topics

Here is a list of available topics.  Enter any topic name to get more help.

ASSERTION           DELETION            LOOPING             SHIFTING
ASSIGNMENT          DICTIONARIES        MAPPINGMETHODS      SLICINGS
ATTRIBUTEMETHODS    DICTIONARYLITERALS  MAPPINGS            SPECIALATTRIBUTES
ATTRIBUTES          DYNAMICFEATURES     METHODS             SPECIALIDENTIFIERS
AUGMENTEDASSIGNMENT ELLIPSIS            MODULES             SPECIALMETHODS
BASICMETHODS        EXCEPTIONS          NAMESPACES          STRINGMETHODS
BINARY              EXECUTION           NONE                STRINGS
BITWISE             EXPRESSIONS         NUMBERMETHODS       SUBSCRIPTS
BOOLEAN             FLOAT               NUMBERS             TRACEBACKS
CALLABLEMETHODS     FORMATTING          OBJECTS             TRUTHVALUE
CALLS               FRAMEOBJECTS        OPERATORS           TUPLELITERALS
CLASSES             FRAMES              PACKAGES            TUPLES
CODEOBJECTS         FUNCTIONS           POWER               TYPEOBJECTS
COMPARISON          IDENTIFIERS         PRECEDENCE          TYPES
COMPLEX             IMPORTING           PRIVATENAMES        UNARY
CONDITIONAL         INTEGER             RETURNING           UNICODE
CONTEXTMANAGERS     LISTLITERALS        SCOPING             
CONVERSIONS         LISTS               SEQUENCEMETHODS     
DEBUGGING           LITERALS            SEQUENCES           

help> 

ask for help on a specific module

help> http      
Help on package http:

NAME
    http

MODULE REFERENCE
    https://docs.python.org/3.8/library/http

    The following documentation is automatically generated from the Python
    source files.  It may be incomplete, incorrect or include features that
    are considered implementation detail and may vary between Python
    implementations.  When in doubt, consult the module reference at the
    location listed above.

PACKAGE CONTENTS
    client
    cookiejar
    cookies
    server

CLASSES
    enum.IntEnum(builtins.int, enum.Enum)
        HTTPStatus
        (....)

you get the idea...


use dir in Python 3 to see what is available on the module

import http

dir(http)

['HTTPStatus',
 'IntEnum',
 '__all__',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'client']

use type to confirm the type of variable

var1 = 'a'
var2 = 1
var3 = ('a','b','c')
var4 = ['a','b','c']
var5={'Website':'AntonioFeijao.com','LinkedIn':'AntonioFeijaoUK'}                     

type(var1)
str

type(var2)
int

type(var3)
tuple

type(var4)
list

type(var5)
dict

Conclusion

On this page I mentioned : * import this * help() * help> modules * help> topics * help> http * dir(imported_module) * type(var...)

I hope these tips will help you explore Python 3 further and at the same time having some fun.

There are, of course various free resources online,
for example this one https://www.tutorialspoint.com/python3/index.htm .

Have fun learning,

Antonio Feijao

AWS run multiple commands in multiple AWS accounts in the AWS Organizations at the same time

AWS run multiple commands in multiple AWS accounts in the AWS Organizations at the same time

** USE AT YOUR OWN RESPONSABILITY **

I needed to run the same AWS CLI command in multiple accounts within an AWS Organization.

I like code that is easy to read, does what is meant to do, is quick to run, is "safe" and efficient.

Might not be perfect for everyone, but it does well what I need it to do, and might be useful for others or for me again in the feature.

So, I built a script to do what I need in a way that can be re-used for multiple commands in multiple accounts at the same time.

If you have, for example +100 AWS Accounts, you might want to "slow down" this script.

for example, I sometimes use the for-loop below to "slow down" my script :)

for NUMBER in $(seq 0 9); do
    ./99-run-this-command-on-this-account.sh ${NUMBER} ; sleep 20
done

Script to run command multiple AWS account within the organisation

Important

cat 99-run-this-command-on-this-account.txt

#!/bin/bash


run_this_command () {
      account_id=$1
    account_name=$2
       role_name=$3

    echo "--------------------------------------------"
    echo "Going to run a command on ${account_id}, ${account_name}, using the role: ${role_name}"

    # assume a role in the account
    new_role=$(aws sts assume-role --role-arn "arn:aws:iam::${account_id}:role/${role_name}" --role-session-name ${account_id}-${role_name})

    AWS_ACCESS_KEY_ID=$(echo ${new_role} | jq -r '.Credentials.AccessKeyId' )
    AWS_SECRET_ACCESS_KEY=$(echo ${new_role} | jq -r '.Credentials.SecretAccessKey' )
    AWS_SESSION_TOKEN=$(echo ${new_role} | jq -r '.Credentials.SessionToken' )

    ACCOUNT_ID=$(echo ${new_role} | jq -r '.AssumedRoleUser.Arn' | cut -f 5 -d ':')

    export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    export AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}

    export ACCOUNT_ID={ACCOUNT_ID}

    echo "Got key_id >>${AWS_ACCESS_KEY_ID}<< for account_id >>${account_id}, ${account_name}<< "

    #aws s3 ls

    aws securityhub update-standards-control \
        --standards-control-arn "arn:aws:securityhub:eu-west-2:${account_id}:control/cis-aws-foundations-benchmark/v/1.2.0/1.13" \
        --control-status "DISABLED" \
        --disabled-reason "SCP applied at the root level for the organisation will block any root actions."

    aws securityhub update-standards-control \
        --standards-control-arn "arn:aws:securityhub:eu-west-2:${account_id}:control/cis-aws-foundations-benchmark/v/1.2.0/1.14" \
        --control-status "DISABLED" \
        --disabled-reason "SCP applied at the root level for the organisation will block any root actions."

    aws securityhub update-standards-control \
        --standards-control-arn "arn:aws:securityhub:eu-west-2:${account_id}:control/aws-foundational-security-best-practices/v/1.0.0/IAM.6" \
        --control-status "DISABLED" \
        --disabled-reason "SCP applied at the root level for the organisation will block any root actions."

    #for bucket in $(aws s3api list-buckets | jq '.Buckets[].Name' | sed s/'"'//g) ; do
    #    echo "${account_id}, ${account_name}, enabling PublicAccessBlockConfiguration for bucket: ${bucket}" ;
    #    #aws s3api get-public-access-block --bucket ${MYBUCKET} | jq '.PublicAccessBlockConfiguration' ;
    #    aws s3api put-public-access-block \
    #          --bucket ${bucket} \
    #          --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
    #done

}




number=$1
role_name=<ROLE_NAME_IN_HERE>
filter_name="prod"

# DO I NEED to filter by account name?
#
# DO I NEED a specific list of accounts?  >> command to list account - 'aws organizations list-accounts'
#
# Below, sample filter to list account by creation date.
# cat ./results/00-list-of-accounts-in-org.txt
# jq -r '.Accounts[] | "\(.JoinedTimestamp);\(.Id);\(.Name)" ' ./results/00-list-of-accounts-in-org.txt  | sort | grep '2022-03' | awk -F ';' '{print $2 ";" $3}'
# jq -r '.Accounts[] | "\(.JoinedTimestamp);\(.Id);\(.Name)" ' ./results/00-list-of-accounts-in-org.txt  | sort | grep '2022-03' | awk -F ';' '{print $2 ";" $3}' > work-on-this-list-of-accounts.txt
#
# pick your for-loop
#
#for account in $(jq '.Accounts[] | .Id + ";" + .Name' ./results/00-list-of-accounts-in-org.txt | tr -d '"' | sort | grep ^${number} | grep -v ${filter_name} ); do
#for account in $(jq '.Accounts[] | .Id + ";" + .Name' ./results/00-list-of-accounts-in-org.txt | tr -d '"' | sort | grep ^${number} ); do
 for account in $(cat ./work-on-this-list-of-acocunts.txt | tr -d '"' | sort | grep ^${number} ); do

    #sample account result: "000123456789;my-super-aws-account-name"
    account_id=$(  echo ${account} | cut -f1 -d ';' )
    account_name=$(echo ${account} | cut -f2 -d ';' )

    echo "Getting ready to run a command on this account : ${account_id}; ${account_name}
        "
    run_this_command ${account_id} ${account_name} ${role_name} &

done

AWS Systems Manager or AWS SSM to create a private Networking tunnel to resources in the private subnet

AWS Systems Manager or AWS SSM to create a private Networking tunnel to resources in the private subnet

Pre-requisit

you need the AWS SSM agent installed on your laptop/desktop - documentation here https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html

AWS SSM create a tunnel to Linux instance in an AWS private subnet

1) Assume a role in the account

2) Get the instance ID you want to tunnel to

3) Start a session with AWS SSM agent

Sample commands for creating a tunnel to the instance on port 3389 and my PC/laptop localhost on port 5222

# Using a variable, so easier to reuse the command

instance_id="i-Xxxxxx"


# Sample command for creating a tunnel to the instance on port 22 and my PC/laptop localhost on port 5222

aws ssm start-session --target ${instance_id} \
    --document-name AWS-StartPortForwardingSession \
    --parameters portNumber="22",localPortNumber="5222" \
    --region eu-west-2

After creating the tunnel to the instance, you still need a valid ssh key to ssh into the ec2 instance.

Via the SSM in the console, you could add your public key to the authorized-keys - where is another website explaning that https://www.ssh.com/academy/ssh/authorized-keys-file


Example using AWS SSM to create a private networking tunnel to use as remote desktop into a Windows instance in a private subnet

Sample command for creating a tunnel to the instance on port 3389 and my PC/laptop localhost on port 5222

Same step above 1) and 2)

3) Create the tunnel with the RDP port as a destination portNumber

# Using a variable, so easier to reuse the command

instance_id="i-Xxxxxx"


# Sample command for creating a tunnel to the instance on port 22 and my PC/laptop localhost on port 5222

aws ssm start-session --target ${instance_id} \
    --document-name AWS-StartPortForwardingSession \
    --parameters portNumber="3389",localPortNumber="5222" \
    --region eu-west-2

Now, using your favourity remote destop application, you can RDP to localhost:5222 which will be tunneled into the Windows instance in the private subnet on port 3389.


Additional documentation

  • "Port Forwarding Using AWS System Manager Session Manager"

https://aws.amazon.com/blogs/aws/new-port-forwarding-using-aws-system-manager-sessions-manager/

  • "... Tunnel through AWS Systems Manager to access my private VPC resources"

https://aws.amazon.com/premiumsupport/knowledge-center/systems-manager-ssh-vpc-resources/

Change video playback speed rate with javascript

Change video playback speed rate with javascript

You can run these commands in the

document.getElementsByTagName("video")[0].playbackRate = '1.00'
document.getElementsByTagName("video")[0].playbackRate = '1.50'
document.getElementsByTagName("video")[0].playbackRate = '1.75'
document.getElementsByTagName("video")[0].playbackRate = '2.00'
document.getElementsByTagName("video")[0].playbackRate = '2.50'

You can also save them as a bookmark link

sample with querySelector('video')

javascript: (function () {    document.querySelector('video').playbackRate = 1.00;})();
javascript: (function () {    document.querySelector('video').playbackRate = 1.50;})();
javascript: (function () {    document.querySelector('video').playbackRate = 1.75;})();
javascript: (function () {    document.querySelector('video').playbackRate = 2.00;})();
javascript: (function () {    document.querySelector('video').playbackRate = 2.50;})();

sample with getElementsByTagName("video")

javascript: (function () {    document.getElementsByTagName("video")[0].playbackRate = 1.00;})();
javascript: (function () {    document.getElementsByTagName("video")[0].playbackRate = 1.50;})();
javascript: (function () {    document.getElementsByTagName("video")[0].playbackRate = 1.75;})();
javascript: (function () {    document.getElementsByTagName("video")[0].playbackRate = 2.00;})();
javascript: (function () {    document.getElementsByTagName("video")[0].playbackRate = 2.50;})();

sample with getElementsByTagName("audio")

javascript: (function () {    document.getElementsByTagName("audio")[0].playbackRate = 1.00;})();
javascript: (function () {    document.getElementsByTagName("audio")[0].playbackRate = 1.50;})();
javascript: (function () {    document.getElementsByTagName("audio")[0].playbackRate = 1.75;})();
javascript: (function () {    document.getElementsByTagName("audio")[0].playbackRate = 2.00;})();
javascript: (function () {    document.getElementsByTagName("audio")[0].playbackRate = 2.50;})();