# Module 5 Storage and Databases ## Instance Stores and Amazon Elastic Block Store (Amazon EBS) * Block Level Storage * to store files (bytes stored on disk) * When data change only change that section * Hard drive * File systems, databases use it * Instance STore Volumes * storage that can be provided by EC2 instance * Physical attached to host that EC2 is running on * If EC2 is terminated, the data on the ISV is deleted * As the EC2 might start up on another host * Useful for temp data, scratch files, data that can be easily recreated without consequence * Not good for persisting data outside of lifecycle of EC2 * Amazon Elastic Block Store (EBS) * Create virtual Hard drives (volumes) that are attached to the EC2 * Not tied to the host * persist data outside of lifecycle of EC2 * Can configure the type you want (size, type) and attach it the EC2 * Can take incremental back ups of data (snapshots) * configurable * only the blocks of data that have chagned are saved * Up to 16TB * SSD by default, but HDD options ## Amazon Simple Storage Service (Amazon S3) * Data to be stored somewhere * Store and retrieve unlimited amount of data * Data stored as objects * each object contains data, metadata and key * metadate = info about data, how its used, object size * key = unique id of object * When object is updated, the whole object is modified * Stored in buckets * Max size of object =5Tb * Used for write once read many * Each object has a url * Can version objects (keep history of object, can rollback if deleted) * Create permissions (visibility, write access) for multiple buckets * Different tiers/classes * Storage classes * Standard * 11 9s of durability (remain intact for one year) * for frequent access * Multiple copies are stored in 3 availability zones * Content distributino * data analytics * Static web hosting * Load all static files (html etc) to S3 and check box to host it as site * Standard Infrequent Access (STandard-IA) * For data that is accessed less frequently, but requires rapid access when needed * backups, disasted recovery files, or long term storage * Audit data, stored for seveal years can be moved to other classes * Lower storage price * higher retrieval price * Multiple copies are stored in 3 availability zones * One Zone Infrequent Access * 1 copy are stored in 1 availability zones * Lower storage price than STandard IA * Saving costs on storage * Can easily reproduce data incase of failure of zone or loss of data * Intelligent Tiering * Data with unknown or changing access patterns * Monthly monitroing and automation fee per object * * Glacier Instant Retrieval * For archived data that needs immediate access * Access time of milliseconds (same perfromance as standard) * Glacier Flexible Retrieval * Low cost storage * Takes 1 minutes to 12 hours to access data * Audit data, stored for several years * Use vaults * Glacier Deep Archive * Lowest cost * Retrieve within 12 to 48 hours * Long retention * aim for 1/2 times a years access * Multiple copies are stored in 3 availability zones * Outposts * Creates buckets on Outposts * Easier to retrieve * Puts it on your on premise site * Lifecycle polices * Setup rules to move data between tiers * ie after x days move to another class * Default polics * haven’t accessed an object for 30 consecutive days, Amazon S3 automatically moves it to the infrequent access tier, S3 Standard-IA. If you access an object in the infrequent access tier, Amazon S3 automatically moves it to the frequent access tier, S3 Standard. * ### EBS vs S3 ## Amazon Elastic File System (EFS) * A type of file storage * a storage server uses block storage with a local file system to organize files. Clients access data through file paths. * Ensures that * access the same data at the same time * Storage can handle the amount of data * scale with increase demand * that backups are taken * data is stored is redundantly * management of servers holding data * EFS handles all this * EFS * Multiple instances can access (read/write) the data in EFS at the same time * Linux file system * regional resource * automatically scales * In different availability zones * allows for concurrent access * difference with EBS * does not scale, once you attach it to EC2 thats it * EBS must be in same availability zone * Can be AWS cloud service or on prem * on prem access with AWS Direct Connect ## Amazon Relational Database Service (RDS) * RDBS useful for data stored that has relationships with other stored data * Data is stored in tables * Tables are defined by schemas * relationships between data in tables is done via a key * Querying is done via standard language such as SQL * so is defining the schema of the tables (definition) * And commands (updates/delets/writes) * Supported DB * postgres * mysql * oracle * sql server * Security * at encryption at rest * encryption in tranist * Migrate database from on prem to aws * Lift and shift migration * Have more control over OS, memory, CPU, storage capacity, etc * Can use Amazon RDBS, a managed db service * supportst the major DB engines * Hardware provisionin * Automated patching * backups * redundancy * failover * disaster recovery * Amazon Aurora * enterprise class rdbs * a more managed db system * mysql or postgres flavours * 5 times faster than mysql and 3 time sfaster than postgres * Reduces unecessary IO * cheaper than other db engines * data is replicated across facilities (6 copies at any one time) * up to 15 read replicas * Continuous back up to S3 * Can be slow, due to the overhead of the queries/commands over several tables * For business analytics, over many tables ## Amazon DynamoDB * Serverless DB * Data stored in tables, as items with attributes * Handles the storage, automatic scaling, stored redundently accross multiple AZ, * millisecond respone time * Dont have to provision, patch or manage servers, or install, maintain or operate software * Does not use sql, does not need to define schema * Useful for data that is not rigid (ie cannot be defined by a scehma) and need high performance * Non relational db * Have simple flexible schemas * Can add/remove attributes to a table at any time * NOt every item must have the same attributes * Store data as key-value pairs * key = items * value = attirbutes * Queries are much simpler * focus on collection of items from one table * Not on queries from multiple tables * leads to quick response time and high scalability * It is purpose built and fits a specific usecase * Most data is used for lookup lists * this can be done via non relational DB rather than sqlDB ## Amazon Redshift * Used for data analysing what happened * Using traditional RDBMS for querying data which is constantly updated * causes performance issues * used for high speed real time ingestion, rather than complex queries over ltos of data * VAriety of data that is spread out has issues with this analytics * USe of data wharehousing * engineered for big data and historical analystics instead of operational analysis * For questions about looking backwards,rather than looking at the current information for current processing (which is what RDBMS is built for) * Redshift * DW that is tuned, resiliant and highly scalable * Nodes can handle mutliple PBytes ## AWS Database Migration Service (AWS DMS) * Help migrate DB onto AWS securly adn easily * Source DB remains fully operational during the migration * reduces the downtime * Dont have to migrate to the same type of DB * Same type migrations = homogenous * straigthforward * source and target are different = hetrogenous * Two step process * Need to convert schema structure/data types and db code using AWS Schema Conversion Tool to match the target db * Then use DMS to migrate the data * Can migrate from from on prem to EC2 or RDS * Other migrations include * dev/test db migrations * copy prod data to test env (one off or continuously) * db consolidations * have multipe db but move to one db * continous db replication * for disaster recovery or geographic separation ## Additional Database SErvices * Amazon DocumentDB * Document db * supports MongoDB * Amazon Neptune * graph DB * works with highly connected datasets * ie recommendation engines, fraud detection, and knowledge graphs. * Amazon Quantum Ledger Database (Amazon QLDB) * review a complete history of all the changes that have been made to your application data. * Data never deletd * Amazon Managed Blockchain * create and manage blockchain networks with open-source frameworks. * Blockchain is a distributed ledger system that lets multiple parties run transactions and share data without a central authority. * Amazon ElastiCache * adds a caching layer on top of db * improve read times for common requests * two types: redis and memcached * Amazon DynamoDB Accelerator * in memory cahce for dynamo db * millis to micro ## Links * * * * * * * * *