If you've ever managed inventory for a retail business, you know the nightmare: the system says you have 5 units, but you can only find 3 on the shelf. Or worse, you sell something the system said was in stock, only to discover it was already gone.

Traditional inventory systems store stock as a single number in a database column — and every operation just increments or decrements that number. It works... until it doesn't. A failed API call, a race condition, a developer forgetting to update the count in one edge case — and suddenly your stock is off. Forever. And you have no idea when or why it went wrong.

I built Lensiro's stock system so that this class of bugs is architecturally impossible.

The Core Idea: Never Store the Stock Count

Here's the fundamental insight: the stock quantity is never stored anywhere. There is no quantity column in the Stock table. Zero. None.

Instead, every time anyone asks "how many of item X do we have at branch Y?", the system calculates it from scratch by summing up every movement that has ever happened to that item.

Stock = SUM(all inflows) - SUM(all outflows)

That's it. That's the whole trick.

The Ledger: Every Movement Tells a Story

Every stock change is recorded as an immutable event in its own dedicated table. There are 8 types of movements:

Things that add stock (+):

• Purchase Receive — goods physically received from a supplier

• Stock Opname (positive) — physical count reveals more stock than expected

• Transfer In — items arriving from another branch

Things that remove stock (-):

• Sale — items sold to customers

• Complimentary — items given away

• Purchase Return — items sent back to supplier

• Transfer Out — items shipped to another branch

• Stock Opname (negative) — physical count reveals less stock than expected

Each movement record contains:

• The count (how many)

• The timestamp (when)

• The cost price (for accounting)

• A link back to its source document (the sale, the purchase order, the transfer)

These records are append-only. Once a purchase receive is recorded, it stays recorded. You don't go back and edit the number — if there's a correction, you create a new adjustment record. The history is sacred.

This Was Not Easy to Implement

The idea is simple — just sum up all movements. The implementation? Not so much. Here's what made it hard and what I did to make it work.

The Query From Hell

The stock count query unions 9 different tables into a single CTE (Common Table Expression). Each sub-query has its own JOIN conditions, its own sign logic (positive for inflows, negative for outflows), and its own edge cases. The first version of this query was slow. Really slow. Loading the stock list page with a few thousand items could take seconds.

The fix was pre-aggregating inside each UNION ALL branch. Instead of dumping every individual row into the CTE and then aggregating at the end, each sub-query does its own GROUP BY stockId and SUM(count) first. This means the CTE combines 9 small, pre-summed result sets instead of potentially tens of thousands of raw rows. The difference was night and day.

I also split the query into layers using multiple CTEs:

1. movements — unions and pre-aggregates all 9 movement sources

2. filtered_stock — applies all the filter conditions (branch, category, brand, color, prescription, price range, name search) against the Stock and Item tables

3. stock_counts — joins filtered stocks with their movements, applies HAVING clauses for min/max stock filters

This way the database can optimize each layer independently. Filters narrow down which stocks we care about before we join with movements, so we're not calculating counts for items nobody asked for.

The Consistency Problem

If you're not careful, event-sourced stock calculation can show inconsistent numbers. Imagine:

• The stock list page calculates stock using the SQL CTE on the server

• A user clicks "Detail" on an item, which fetches all relations and calculates stock using TypeScript on the client (calcStockPlus() minus calcStockMinus())

• If the logic in SQL and TypeScript doesn't match exactly, the numbers disagree

This actually happened during development. I added a new movement type (StockBuffer) to the SQL query but forgot to add it to the TypeScript calculation. The stock list showed one number, the detail view showed another. Staff would understandably lose trust in the system.

The solution was discipline: every time a new movement type is added, it must be added in three places — the SQL CTE, the TypeScript calcStock functions, and the generateTrail function that renders the timeline. There's no clever abstraction that enforces this automatically. It's just a rule you follow.

FIFO Cost Price Calculation

Getting the stock count right was only half the battle. For accounting, every outflow (sale, return, transfer) needs a cost price — how much did we pay for this specific unit? We use FIFO (First In, First Out): the oldest purchased units are assumed to be sold first.

The tricky part: what happens when you sell an item before the purchase is recorded? This happens in real retail — you receive goods on Monday, sell one on Tuesday, but the purchase order isn't entered until Wednesday. The outflow happened before the inflow exists in the system.

I handle this with retroactive cost pricing. Outflows that happen before sufficient inflows are queued as "pending." When a new purchase is recorded, the system re-processes pending movements and assigns them the correct cost price. This runs asynchronously using waitUntil() after the main transaction commits, so it doesn't block the user.

Batch Operations

Early on, creating stock records one at a time during bulk operations (like receiving a purchase order with 50 line items) was painfully slow — each one triggered a separate findFirst then create query. I replaced this with fetchOrCreateStocksBatch(), which finds all existing stocks in one query, identifies the missing ones, bulk-creates them with createMany, and fetches the new ones. What used to be 100+ queries became 3-4.

Why "Cannot Be Wrong"?

When I say this system "cannot be wrong," I mean something very specific. The stock count is mathematically derived from the complete set of transactions. If the transactions are recorded correctly, the stock count is correct — by definition. There is no way for the count to "drift" out of sync, because there is no separate count to drift.

Here's what makes this different from a traditional UPDATE stock SET quantity = quantity - 1 approach:

1. No Stale State

There is no cached or stored quantity that can go stale. Every query recalculates from the source of truth. If you run the query right now and again in 5 minutes (with no new transactions), you get the exact same answer.

2. Complete Auditability

Every single unit of stock can be traced back to how it got there. Got 12 units? You can see: 10 came from a purchase on January 3rd, 3 came from a transfer on January 15th, and 1 was sold on January 20th. 10 + 3 - 1 = 12. The math is right there on screen.

3. Self-Correcting with Stock Opname

When staff does a physical stock take (we call it "Stock Opname"), they count what's actually on the shelf and enter it. The system compares: ledger says 12, physical count says 11. It creates an adjustment record of -1. Now the ledger says 11. The discrepancy is recorded as an event itself — not silently patched.

4. Atomic Transactions

Every operation that touches stock is wrapped in a database transaction. If you're recording a sale with 3 line items, either all 3 items get their movement records, or none of them do. No partial states.

5. Branch Isolation

Each branch has its own stock ledger. Branch A's inventory is completely independent of Branch B's. When items move between branches, it creates a pair of records: a decrease at the source branch and an increase at the destination branch (only when the receiving branch confirms receipt).

The SQL Behind It

On the backend, a single SQL query with a Common Table Expression (CTE) unions all movement tables together and sums them up:

WITH movements AS (
  SELECT stockId, SUM(count) as value
  FROM PurchaseReceiveItem
  GROUP BY stockId

  UNION ALL

  SELECT stockId, SUM(count) as value
  FROM StockAdjustment
  GROUP BY stockId

  UNION ALL

  SELECT targetStockId, SUM(count) as value
  FROM StockMutationItem
  WHERE received = true
  GROUP BY targetStockId

  UNION ALL

  SELECT stockId, -SUM(count) as value
  FROM SaleItem
  GROUP BY stockId

  -- ... and so on for all movement types
)
SELECT stockId, COALESCE(SUM(value), 0) AS stockCount
FROM movements
GROUP BY stockId

On the client side, the same logic is mirrored in TypeScript — calcStockPlus() sums all inflows, calcStockMinus() sums all outflows, and calcStock() returns the difference. Both server and client use the same formula. There's nowhere for the numbers to disagree.

The Trail View: Show Your Work

My favorite part of this system is the trail view. When a staff member clicks on any stock item, they see a full chronological timeline of every movement, each one clickable back to the source document.

It's like a bank statement for inventory. Every "deposit" and "withdrawal" is listed. The running total is visible. Anyone can verify the math by scrolling through the list.

This isn't just for debugging — it's a daily tool. When a staff member questions the count, they don't need to call IT. They click "Detail," read the trail, and understand exactly why the system says what it says.

But Wait — It Can Still Be Bugged

I want to be honest here. When I say the system "cannot be wrong," I'm talking about the architecture — the structural impossibility of stock count drift. The system is designed so that if all movements are recorded, the count is mathematically guaranteed to be correct.

But software is software. Bugs can still happen:

• A movement could fail to be recorded. If a purchase receive action crashes halfway through and the transaction doesn't roll back properly, a movement might be lost. The stock count would then be wrong — not because of drift, but because of missing data.

• A movement could be recorded with the wrong count. If the UI sends count: 10 instead of count: 1 due to a frontend bug, the ledger faithfully records the wrong number. Garbage in, garbage out.

• A new movement type could be added without updating the calculation. If a developer adds a 9th movement type but forgets to include it in the calculation functions or the SQL query, those movements would be invisible.

• Concurrent transactions could interact unexpectedly. Although we use database transactions, extremely high concurrency on the same stock item could theoretically cause issues depending on the isolation level.

• The client and server calculations could diverge. We mirror the logic in both SQL and TypeScript. If someone updates one but not the other, you'd see different numbers depending on where you look.

The architecture eliminates the most common class of inventory bugs — the silent drift that accumulates over months and is impossible to diagnose. But it doesn't make the software immune to all bugs. No architecture can.

What it does give you is debuggability. When something looks wrong, you can open the trail, read every movement, and find the problem. Compare that to a traditional system where the stock is just... a number. How did it get there? Who knows.

The Buffer System: Handling In-Progress Sales

One more detail I'm proud of. In optical retail, there's often a gap between selling an item and physically handing it to the customer (because lenses need to be fitted into frames). During this gap, the item is "sold" but still physically in the store.

We handle this with a buffer concept. The stock count includes sold-but-unfulfilled items as decreases, but the UI shows a clear breakdown:

• Total Stock: 5 (the ledger total)

• Buffer: 2 (sold but not yet fulfilled)

• Available: 3 (actually available to sell)

This prevents double-selling without hiding information from staff.

Lessons Learned

1. Derived state > stored state. If you can calculate it, don't store it. Stored state lies. Calculated state can't.

2. Make the audit trail a feature, not an afterthought. When staff can see why the number is what it is, trust in the system goes up dramatically.

3. Event sourcing isn't just for distributed systems. Even a simple retail app benefits from treating every state change as an immutable event.

4. The best debugging tool is transparency. When something goes wrong, a complete trail of events is worth more than any amount of logging.

About Lensiro

Lensiro is a complete retail management platform built specifically for optical stores. It handles everything an eyewear business needs in one place — from inventory and stock management across multiple branches, to point-of-sale, purchasing, member management, and accounting with double-entry bookkeeping. If you run an optical retail business and want a system that actually gets inventory right, check it out at lensiro.com.

Built with Next.js, PostgreSQL, Prisma, and a lot of stubbornness about doing inventory right.

ネタバレ: あるほこりだらけの会議室から始まり、最後は午前3時にオフィスのネットワークを総配線し直すことになりました。

フェーズ1：眠れるMacBookに息を吹き込む

オフィスの棚には、誰も使っていない M1 MacBook Pro（32 GB RAM）が12台並んでいました。時間がたてば資産価値は下がるだけ――そこで私は「これをLLM推論サーバーにしよう」と提案。CEOはコスト削減のアイデアを気に入り、私たちはすぐに行動を開始しました。

スタック構成

特別なものはありません。

• Ollama でモデル提供（4つ試した中で最終的に採用）

• 1台のMacBookに HAProxy を入れ、シンプルなラウンドロビン負荷分散

• Prometheus + Grafana でメトリクスとダッシュボード

• 6 m²の会議室が“データセンター”なので、冷却用の安い卓上ファン

• MacBookは1台ずつ設置

フレームワーク比較

1週間で12台のMacBookが30Bモデルをローカルで動かし、本番トラフィックの約5 %を処理できるようになりました。

フェーズ2：Mac Studioの寄り道 ― “大きい＝速い”とは限らない

成功すると欲が出るもの。マネジメントは「メモリが多ければスループットも上がる」と考え、Mac Studio（512 GB RAM・80-core GPU）を6台購入。各Mac StudioでMacBook 8台分の性能を期待しました。

しかし現実は、LLMの速度はほぼGPUコア数に比例し、Amdahlの法則が示すようにパイプラインの一部は並列化できません。その結果、Mac Studio 1台はMacBookの約3–4倍しか速くありませんでした（8倍ではない）。

教訓は得ましたが、6台のMac Studioでトラフィックカバー率は約25 %に増えました。

フェーズ3：200台のMac miniとAnsibleの歓喜

なぜMac mini？

コスト分析の結果、Mac mini（20-core GPU）2台で1台のMac Studioよりもトークンあたりのコストパフォーマンスが高いと判明。200台を一括購入しました。

自動プロビジョニング

200台を手動セットアップなど論外。そこでAnsibleを導入しました。

（以下は例。本番環境はもっと複雑です）

# プレイブックの抜粋
- hosts: mac
  tasks:
    - name: Install Ollama
      homebrew:
        name: ollama
        state: present
    - name: Configure model
      shell: ollama pull mistral:7b-instruct
    - name: Register with HAProxy
      template:
        src: haproxy.cfg.j2
        dest: /usr/local/etc/haproxy/haproxy.cfg

最初の50台が一気に稼働した瞬間はまさに魔法のよう。あとは繰り返しでOKです。

午前3時のネットワーク大混乱

最大の難所はネットワークでした。サーバー群用に専用VLANが必要でしたが、手元のYamahaルーターのマニュアルは中途半端な翻訳PDF、前任のネットワーク技術者は退職済み。ポート設定を1つミスっただけでオフィスWi-Fiが全滅。12時間、コーヒー3ポット、VLANタグ付けの猛勉強の末、無事に両ネットワークが復旧しました。

今後の計画

ラック化スケール – 200台は収まっているものの、エアフローとケーブル管理が限界。次は42Uラックと本格PDUを導入予定です。

パート2ではVLAN奮闘記の詳細と、クラスターを支えるGrafanaダッシュボードを紹介します。お楽しみに！

読んでくれてありがとう！

質問やコメント、あなたの失敗談などあればぜひ教えてください。情報交換しましょう。

Spoiler: it started with a dusty meeting room and ended with me rewiring the entire office network at 3 a.m.

Phase One: Breathing Life into Unused MacBooks

Our office shelves were lined with twelve M1 MacBook Pros (32 GB RAM) that nobody had touched in months. Instead of letting them depreciate silently, I proposed re‑purposing them as LLM inference servers. The CEO loved the cost‑saving angle, so we rolled up our sleeves.

The stack

Nothing fancy:

• Ollama for model serving (picked after trying four frameworks—see below)

• HAProxy on one MacBook for simple round‑robin load balancing

• Prometheus + Grafana for metrics and dashboards

• A cheap desktop fan to keep the “data center” (a 6 m² meeting room) cool

• Installing one macbook at atime

Framework bake‑off

Within a week we had twelve MacBooks serving local 30B models and handling ~5 % of live traffic.

Phase Two: The Mac Studio Detour—Bigger Isn’t Always Better

Success breeds ambition. Convinced that “more memories = more throughput,” management ordered six fully loaded Mac Studio (512 GB RAM, 80‑core GPU) machines, expecting each to replace eight MacBooks.

Reality check: LLM speed scaled almost linearly with GPU cores, not RAM, and Amdahl’s Law reminded us that some parts of the pipeline stay serial no matter what. A single Mac Studio was only about 3×–4x faster than a MacBook, not 8×.

Lesson learned, but the six Mac Studios still bumped us to ~25 % traffic coverage.

Phase Three: 200 Mac Minis and the Joy of Ansible

Why Mac minis?

A cost analysis showed that two Mac minis (20‑core GPU) delivered more tokens per dollar than a single Mac Studio. We bulk‑ordered two hundred of them.

Automated provisioning

Hand‑installing 200 machines was a non‑starter, so I dove head‑first into Ansible:

(Below is example. the real deal is much complex)

# excerpt from playbook
- hosts: mac
  tasks:
    - name: Install Ollama
      homebrew:
        name: ollama
        state: present
    - name: Configure model
      shell: ollama pull mistral:7b-instruct
    - name: Register with HAProxy
      template:
        src: haproxy.cfg.j2
        dest: /usr/local/etc/haproxy/haproxy.cfg

Bringing the first 50 nodes online felt like magic. After that, it was rinse‑and‑repeat.

The 3 a.m. network meltdown

The real nightmare was networking. We needed a separate VLAN for the “server farm,” but the only documentation for our Yamaha router was a half‑translated PDF, and the prior network engineer had left months earlier. One mis‑tagged port later, the office Wi‑Fi went dark. Twelve hours, three pots of coffee, and a crash course in VLAN tagging later, both networks were humming.

Future Plans

Scale to racks – 200 minis fit, but airflow and cabling are becoming a headache. A 42U rack with proper PDUs is next.

Stay tuned for part 2; next time I’ll cover the VLAN saga in detail and share the Grafana dashboards that keep this Frankencluster alive.

Thanks for reading!

Questions, comments, or horror stories of your own? Let me know below; I’d love to compare notes.

Among Snowflake's myriad features, User-Defined Functions (UDFs) and Procedures stand out. UDFs are functions that you can call within a SELECT (or other) query to format data to your specifications. Procedures, on the other hand, are functions that you can call to process tasks in the background, including creating, updating, and deleting records.

However, there was no way to manage, test, or review these functions before deploying them to production—until Snowflake implement integration with git. Although it's still in open preview, this integration has the potential to solve that problem effectively.

Prepare Codebase

Let's start by preparing the codebase. Here's the directory structure:

As you can see, I'm using Poetry to set up the virtual environment and manage packages. I also employ Ruff for linting, ensuring that my code remains clean and maintainable (well, one can hope (´ω`)).

The source code resides in the /src directory, which I’ve divided into /src/procedures and /src/udf. For this example, I'll create a simple fetch procedure that, given a table name, returns ten rows from it.

import os

from snowflake.snowpark import Session, Table


def fetch(session: Session, table_name: str) -> Table:
    return session.table(table_name).select(["id"]).limit(10)

if __name__== "__main__":
    from snowflake.snowpark import Session

    session = Session.builder.configs({
        "account": os.getenv("SNOWFLAKE__ACCOUNT"),
        "user": os.getenv("SNOWFLAKE__USER"),
        "password": os.getenv("SNOWFLAKE__PASSWORD"),
        "role": os.getenv("SNOWFLAKE__ROLE"),
        "warehouse": os.getenv("SNOWFLAKE__WAREHOUSE"),
        "database": os.getenv("SNOWFLAKE__DATABASE"),
        "schema": os.getenv("SNOWFLAKE__SCHEMA"),
    }).create()

    table = fetch(
        session=session,
        table_name="TEST_TABLE",
    )

    print(table.collect())

This simple example demonstrates how to fetch data from a Snowflake table using a stored procedure, managed and versioned in a GitHub repository. This code enables you to test it locally by running poetry run python3 src/procedures/fetch/fetch.py.

For the full code and additional examples, visit my GitHub repository.

Create Github Personal Access Token

We've written code for Snowflake to understand! Yay! But now we need Snowflake to read our code. To enable this, we need to generate a personal access token (PAT) that allows Snowflake to access our GitHub repository.

Here is the step:

1. Open the GitHub page and click on your profile picture in the top right corner.

2. 

   Click on Settings.

3. 

   In the sidebar, click on Developer settings.

4. 

   Go to Personal access tokens → Fine-grained tokens.

5. 

   Generate a new token. Select your repository and set the expiry date.

6. 

   Grant it permission to read contents.

7. 

   Done!

You should now have your GitHub PAT, which will look something like this:

github_pat_xxxxxxxxxxxx....

Create Needed Snowflake Resources

Okay, let's jump into the Snowflake console and get our hands dirty. First, we need to save our GitHub Personal Access Token in Snowflake as a Secret.

CREATE OR REPLACE SECRET git_secret
    TYPE = password
    USERNAME = 'dummy-username'
    PASSWORD = 'github_pat_xxxxxxxxxxxx....';

The username doesn't really matter; you can input whatever you want. The password must be the generated GitHub PAT token.

Next, using that Secret, let's create our API Integration to integrate GitHub and Snowflake.

CREATE OR REPLACE API INTEGRATION git_api_integration
  API_PROVIDER = git_https_api
  API_ALLOWED_PREFIXES = ('https://github.com/alvinend')
  ALLOWED_AUTHENTICATION_SECRETS = (git_secret)
  ENABLED = TRUE;

Using that API Integration, let's create the Git repository.

CREATE OR REPLACE GIT REPOSITORY snowflake_extensions
  API_INTEGRATION = git_api_integration
  GIT_CREDENTIALS = git_secret
  ORIGIN = 'https://github.com/alvinend/snowflake-extensions.git';

Invoke fetch to fetch the repository.

ALTER GIT REPOSITORY snowflake_extensions FETCH;

Check if our Snowflake Git repository can actually read it.

LS @snowflake_extensions/branches/main;

Now, let's declare our procedure. This simple procedure accepts a table name and returns ten rows from that table.

CREATE OR REPLACE PROCEDURE fetch(
    table_name VARCHAR,
)
  RETURNS TABLE(
    ID VARCHAR,
    NAME VARCHAR
  )
  LANGUAGE PYTHON
  RUNTIME_VERSION = '3.11'
  PACKAGES = ('snowflake-snowpark-python')
  IMPORTS = ('@temp_snowflake_extensions/branches/main/src/procedures/fetch/fetch.py')
  HANDLER = 'fetch.fetch';

All that's left to do is call it!

CALL fetch(
    'TEST_TABLE'
);

Closing

Managing Snowflake's UDFs and procedures with GitHub can significantly streamline your development process, enabling collaborative work and ensuring better code management. By leveraging Snowflake's powerful data platform and integrating it with GitHub, you can easily version-control your UDFs and procedures, reducing the risk of errors and making your workflows more efficient.

In this blog, we walked through the entire process—from creating a GitHub personal access token to integrating it with Snowflake. By following these steps, you can ensure that your Snowflake environment is robust, scalable, and easy to manage.

Happy coding, and may your data always be clean and your queries always be fast!

In some cases, we want to deploy our trained deep learning models or pre-trained models from platforms like Hugging Face to AWS Lambda for serverless inference.

While the official service for such tasks is AWS Sagemaker, it can sometimes be overly complex for simple deployment needs. Although Sagemaker offers benefits like model management and MLOps, there are scenarios where a simpler solution is preferred.

Deploying large models with just Lambda presents challenges due to the service's size limitations—50 MB for direct uploads (zipped) and 250 MB (unzipped). Using Lambda Docker with ECR can support up to 10 GB, but storing all files in memory can lead to slower cold starts and increased costs.

To achieve efficient deployment, I recommend using Lambda with EFS as the file system.

How it Works

EFS is a file system that Lambda can access if it is mounted properly. To achieve this, both the EFS resource and Lambda need to be within a VPC.

According to AWS best practices, only resources that need to be accessible from outside the VPC should be placed in public subnets. Typically, this includes NAT gateway, which adds an extra layer of network security by ensuring critical resources are only accessible from within the VPC.

Consequently, EFS mounts should be placed inside private subnets. Since Lambda functions do not have public IPs, they should also be placed in private subnets.

Here is an overview of the steps.

1. Create a Lambda function in a VPC with a private subnet.

2. Create an EFS in the same VPC as the Lambda function, also in a private subnet.

3. Create an EFS Mount Target in the Availability Zones (AZs) where the Lambda function will be deployed.

4. Create a Security Group to enable the Lambda function to access the EFS.

5. Mount the EFS from the Lambda settings.

To insert the file, I usually create a temporary EC2 instance and mount the EFS on that EC2 instance so that I can transfer my files through the EC2 instance to the EFS.

As a prerequisite, I assume that we already have a VPC with a NAT Gateway and internet access set up from the private subnet.

Sounds Hard Enough... How to build it?

There are many ways to build this infrastructure, and I generally recommend using some form of Infrastructure as Code (IaaC) such as Terraform, CloudFormation, or AWS SAM.

Since we are dealing with Lambda, I will be using AWS SAM in this tutorial. If you already use Terraform in your stack, I suggest managing EFS and EC2 with Terraform and Lambda with AWS SAM.

However, for simplicity, I will be using 100% AWS SAM for this tutorial. The easiest way for us to jump-start this is by using sam init.

sam init

After that, you should see the prompt below. Choose 1 - AWS Quick Start Templates .

Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice:

There are a lot of templates. Choose 14 - Lambda EFS example1

Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Data processing
        3 - Hello World Example with Powertools for AWS Lambda
        4 - Multi-step workflow
        5 - Scheduled task
        6 - Standalone function
        7 - Serverless API
        8 - Infrastructure event management
        9 - Lambda Response Streaming
        10 - Serverless Connector Hello World Example
        11 - Multi-step workflow with Connectors
        12 - GraphQLApi Hello World Example
        13 - Full Stack
        14 - Lambda EFS example
        15 - DynamoDB Example
        16 - Machine Learning

Pick your Python version.

Which runtime would you like to use?
        1 - python3.9
        2 - python3.8
        3 - python3.12
        4 - python3.11
        5 - python3.10

By this point, we should have all we need to deploy Lambda and EFS to the cloud.

The complete generated template can be found here.

SAM Init, Lambda EFS example. Generated Template

Okay, Problem solved?

Not exactly. While using sam init can give us a jump start for our development, there are two problems when using it for Lambda with EFS.

1. It doesn't provide a way to put data on the EFS.

2. The bigger problem is that we cannot exactly use it locally. While there are issues on GitHub discussing this, there is currently no way to test it locally.

In the next sections, we will try to fix the above problems.

Improve SAM Template & Add EC2

When adding an EC2 instance, it's not necessary to rewrite the entire template. The template generated by sam init can be hard to read and understand, so I've rewritten it to make it clear what inputs are needed and what resources are required to create resources like EFS, Lambda, and EC2.

Also, to fix the second problem, we need our Lambda to be image-based instead of a zip file.

First, let's start with the parameters:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Build Lambda with EFS!

Parameters:
  VpcId:
    Type: String
    Default: your-vpc-id

  VpcCidr:
    Type: String
    Default: 12.1.0.0/16

  PublicSubnetId:
    Type: String
    Default: your-vpc-id's public subnet

  PrivateSubnetId:
    Type: String
    Default: your-vpc-id's private subnet

Resources:
  # We Fill this Later

As you can see, we need the VPC ID, VPC CIDR, one public subnet ID, and one private subnet ID. Using these inputs, we can create our resources.

Lambda Resources

Next, we define the Lambda resources:

Resources:
  #
  # Lambda
  #
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: [lambda.amazonaws.com]
            Action: ['sts:AssumeRole']
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - ec2:CreateNetworkInterface
                  - ec2:DescribeNetworkInterfaces
                  - ec2:DeleteNetworkInterface
                Resource: '*'
              - Effect: Allow
                Action:
                  - elasticfilesystem:ClientMount
                  - elasticfilesystem:ClientWrite
                Resource: '*'

  LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      FunctionName: my-function-name
      Role: !GetAtt LambdaExecutionRole.Arn

      # Compute
      Timeout: 600
      MemorySize: 4096
      Architectures:
        - x86_64

      # Network
      FileSystemConfigs:
        - Arn: !GetAtt EFSAccessPoint.Arn # TODO: Implement
          LocalMountPath: /mnt/files
      VpcConfig:
        SecurityGroupIds:
          - !Ref EFSAccessSecurityGroup # TODO: Implement
        SubnetIds:
          - !Ref PrivateSubnetId
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./src
      DockerTag: test

To create the Lambda function, we need two resources: the Lambda itself and an IAM Role for executing it (accessing EFS, putting logs, etc.).

EFS Resources
Next, we create our EFS resources:

  #
  # EFS
  #
  EFSAccessSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Security Group for Lambda and EFS communication
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049  # NFS port used by EFS
          ToPort: 2049
          CidrIp: !Ref VpcCidr

  EFSFileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      Encrypted: false

  EFSMountTarget:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFSFileSystem
      SubnetId: !Ref PrivateSubnetId
      SecurityGroups:
        - !Ref EFSAccessSecurityGroup

  EFSAccessPoint:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref EFSFileSystem
      PosixUser:
        Uid: "1000"
        Gid: "1000"
      RootDirectory:
        CreationInfo:
          OwnerGid: "1000"
          OwnerUid: "1000"
          Permissions: "0777"

The above defines the bare minimum so that our EFS can work. We need an EFS File System, a mount target, and an access point. Additionally, we need a Security Group to allow access between EFS and Lambda.

EC2 Resources

Finally, we add the EC2 instance to interact with EFS:

  #
  # EC2 Instance
  #
  EC2InstanceIAMRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonElasticFileSystemClientFullAccess
        - arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess

  EC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref EC2InstanceIAMRole

  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-07c589821f2b353aa # ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20231207
      InstanceType: t2.micro
      SubnetId: !Ref PublicSubnetId
      SecurityGroupIds:
        - !Ref LambdaSecurityGroup
      IamInstanceProfile: !Ref EC2InstanceProfile

(AMI ID might need to be updated based on the region and availability)

This allows the EC2 instance to access EFS. Additionally, you will need to mount the EFS to the EC2 instance before accessing it. You can find the mounting guide in the AWS official documentation.

The final template can be confirmed at my Github repository.

Bigger Problem: Local Invocation.

While the inability to test locally is not the end of the world, it is certainly a major inconvenience. Without local invocation, you need to deploy every time you want to test your code. This can take hours of your time and has certainly taken hours or even days of mine.

Worry no more, I have found a solution by reverse-engineering sam-cli.

The way it works is that when you press sam local invoke, the sam-cli creates an image, makes a container of it, opens an endpoint, calls it, and then removes the container. So to solve this, we just need to create a container with a Docker volume attached to it, the same directory that we attach our EFS to. In this case, we define it as:

LocalMountPath: /mnt/files

To make this happen, we build our SAM project:

sam build --cached

Using the built image, we run the container with the volume attached to it:

docker run \
    --rm \
    -p 8000:8080 \
    --platform linux/amd64 \
    -v $$(pwd)/efs:/mnt/efs \
    -e AWS_LAMBDA_FUNCTION_MEMORY_SIZE=8192 \
    -e AWS_LAMBDA_FUNCTION_TIMEOUT=600 \
    -e AWS_LAMBDA_FUNCTION_NAME=my-function-name \
    -e AWS_ACCESS_KEY_ID=dummy \
    -e AWS_SECRET_ACCESS_KEY=dummy \
    lambdafunction:test

After that, you can invoke your function with the following command:

curl \
    -X POST \
    http://localhost:8000/2015-03-31/functions/function/invocations \
    -d '{"test":"test"}'

This approach allows you to test your Lambda function locally, saving you significant time and effort.

Closing

By improving our SAM template and adding the necessary configurations, we've streamlined the deployment process for Lambda and EFS resources. We've also implemented a solution for local invocation, significantly reducing development and testing time. Now, you can store your model in EFS and read it from Lambda, enabling efficient handling of large files for serverless inference.

Thank you for following along, and happy coding!

Breakdown

Security

Aurora PostgreSQL has supported Kerberos Auth for a long time. As for MySQL, it appears to have been recently included, as detailed in Using Kerberos authentication for Aurora MySQL. Unfortunately, I could not ascertain the exact date this feature was added.

Monitoring

Much like AWS RDS, both Aurora's MySQL and PostgreSQL support Performance Insights and Enhanced Monitoring features.

Both can also execute Data Activity Streams, essentially logs that record database activity. Examples of such activities include selecting commands against a database, connecting logs, and configuration changes.

However, only PostgreSQL supports Synchronous mode.

Furthermore, there is an Advanced Auditing feature for Aurora MySQL. This feature facilitates the collection of database activity logs and operates with high performance.

Availability

Backtracking "rewinds" the DB cluster to the time you specify. Simple as it may sound, it is a powerful feature. Traditionally (a.k.a AWS RDS MySQL) to retrieve lost data we need to start the instance from backups and get data from there. Unfortunately, it only supports MySQL at the time of writing.

Fault Injection is a way to "sabotage" your instance. Why do we do this? Mainly test the failover capability of our cluster. It supports both Aurora's MySQL and PostgreSQL.

Fast DDL or Instant DDL depending on Aurora MySQL versions, is DDL on steroids. Creating, Altering etc can be done faster with this feature. It supports only Aurora MySQL.

The parallel Query feature is a way to improve I/O. It supports Aurora Mysql

Aurora MySQL supports normal replica, but MySQL support normal replica, cross-region replica and multi-Master Cluster (Replica that can accept write request)

Others

Aurora Serverless like Aurora we can use without managing instances, v2 have more features that v1 doesn't have like AWS Identity and Access Management (IAM) database authentication, and Performance Insights. While we should use v2 if we can it doesn't support Aurora PostgreSQL at the of writing.

Aurora Global Database is a feature that allows replication to cluster in cross-region and it supports MySQL and PostgreSQL.

Aurora lab mode is used to enable Aurora features that are available in the current Aurora database version but are not enabled by default. Only supports Aurora Mysql.

Babelfish for Aurora PostgreSQL is a new capability for Amazon Aurora PostgreSQL-Compatible Edition that enables Aurora to understand commands from applications written for Microsoft SQL Server.

Conclusion

In conclusion, while both Aurora MySQL and PostgreSQL offer a multitude of robust features, there are distinct differences that could influence your choice between the two. When selecting a database, you must carefully consider the unique requirements and constraints of your use case. Features such as Backtracking, Fast DDL, and Advanced Auditing can be pivotal for certain applications and are thus only available for Aurora MySQL. Conversely, the synchronous mode in Data Activity Stream and Babelfish are exclusive to PostgreSQL. Therefore, understanding these differences can guide you in making a more informed choice, ensuring that you leverage the best of what AWS has to offer.

Disclaimer

This document is intended as a general guide to the features of Amazon Aurora within AWS RDS as of the time of writing in July 2023. While every effort has been made to ensure accuracy, the rapidly evolving nature of cloud services means that some information may become outdated or inaccurate over time.

The document does not constitute professional advice, and decisions should not be made solely based on this content. It is recommended that readers refer to the latest official AWS documentation, or consult with an AWS certified professional or the AWS support team for the most accurate and up-to-date information.

Feel free to correct me if I made any mistakes :)

Yet, as I ventured further into my studies, I quickly realized that grasping the full range of offerings across the different flavors of RDS was no small feat. The variety is astounding - MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server, each with its own set of strengths and functionalities. It became apparent that a comprehensive comparison of these systems, highlighting what each one can do, would be an invaluable resource not just for me, but for anyone on a similar path.

This sparked the idea to create this blog post, a concentrated effort to demystify the capabilities of the different RDS systems. The aim was to dissect the key aspects of each RDS flavor, focusing on categories like Backup, Storage, Security, Read Replica, Monitoring, and others, and present a clear, concise, and useful comparison that could serve as a go-to guide for AWS RDS.

Breakdown

General

The above table gives a quick comparison of the AWS RDS offerings, indicating whether they support Parameter Group (PG) and Option Group (OG).

Availability

The above table showcases the high availability options supported by each RDS engine.

Automatic Backup, Manual Backup and Multi-AZ

All systems in RDS support automatic backup, manual backup (Snapshots) and Multi-AZ for additional availability.

Database Mirroring and Always on for SQL Server

Amazon RDS supports Multi-AZ deployments for Microsoft SQL Server by using either SQL Server Database Mirroring (DBM) or Always On Availability Groups (AGs).

When setting up SQL Server Multi-AZ in RDS, it automatically configures all databases on the instance to use DBM or AGs. Amazon RDS manages the primary, the witness, and the secondary DB instance for you. Because the configuration is automatic, RDS selects DBM or Always On AGs based on the version of SQL Server that you deploy.

Refer to the following official documentation for more details:

• Amazon RDS - SQL Server Multi-AZ

Security

Using SSL on MySQL and MariaDB

Both MySQL and MariaDB support the use of SSL/TLS connections for specific user accounts. For example, you can use one of the following statements:

ALTER USER 'encrypted_user'@'%' REQUIRE SSL;

Users can then connect over SSL with the following commands:

• MySQL 5.7+:

mysql -h mysql–instance1.123456789012.us-east-1.rds.amazonaws.com --ssl-ca=global-bundle.pem --ssl-mode=VERIFY_IDENTITY -P 3306 -u myadmin -p

• MySQL < 5.7:

mysql -h mysql–instance1.123456789012.us-east-1.rds.amazonaws.com --ssl-ca=global-bundle.pem --ssl-verify-server-cert -P 3306 -u myadmin -p

• MariaDB:

mysql -h mysql–instance1.123456789012.us-east-1.rds.amazonaws.com --ssl-ca=global-bundle.pem --ssl-mode=REQUIRED -P 3306 -u myadmin -p

Refer to the official documentation for more details:

• MariaDB SSL connections

• MySQL SSL connections

Using SSL on PostgreSQL and SQL Server

For PostgreSQL and SQL Server, you can set rds.force_ssl to 1 in the parameter group to require SSL. By default, this setting is off (0) in SQL Server and PostgreSQL versions below 15. It's on (1) in PostgreSQL version 15 and above.

Refer to the official documentation for more details:

• PostgreSQL SSL

• SQL Server SSL

Using SSL on Oracle

To enable SSL encryption for an Oracle DB instance, add the Oracle SSL option to the option group associated with the DB instance.

Refer to the official documentation for more details:

• Oracle SSL

Using Transparent Data Encryption on SQL Server and Oracle

Amazon RDS supports TDE to encrypt stored data on your DB instances running Microsoft SQL Server or Oracle. TDE automatically encrypts data before it is written to storage and automatically decrypts data when it is read from storage.

Refer to the official documentation for more details:

• SQL Server TDE

• Oracle TDE

Read Replica

Second Tier Read Replica for MySQL

By creating a second-tier Read Replica, you can potentially distribute some of the replication load from the master database instance to a first-tier Read Replica.

AWS RDS allows the creation of 5 first-tier read replicas. Each read replica can create another 5 replicas. Thus, a total of 30 replicas can exist with one master using this feature. However, note that second-tier read replicas may experience higher replication lag.

Refer to this article for more details.

Cross Region Read Replica

AWS RDS can create cross-region read replicas on any system. Notably, SQL Server began supporting cross-region read replicas recently.

Refer to this document for more details.

Write on Read Replica in MySQL and MariaDB

To enable write operations on read replicas in MySQL and MariaDB, set the read_only parameter to false for the DB parameter group associated with your DB instance. Other Amazon RDS engines, such as Amazon Aurora, do not permit the modification of the read_only parameter.

Refer to this article for more details.

External Database Read Replica with MySQL and MariaDB

Replication can be set up between an RDS for MySQL or MariaDB DB instance and a MySQL or MariaDB instance that is external to Amazon RDS using binary log file replication.

Refer to this document for more details.

Backup on Read Replica in MySQL and MariaDB

Automatic backups and manual snapshots are supported on RDS for MySQL, RDS for Oracle or RDS for MariaDB read replicas.

For RDS for PostgreSQL, you can create a manual snapshot of read replicas. However, automated backups for read replicas are only supported for RDS for PostgreSQL 14.1 and higher versions. If you're using RDS for PostgreSQL versions earlier than 14.1 and want a backup, create a snapshot from a read replica.

Refer to this document for more details.

Monitoring

Common Metrics and Enhanced Monitoring

Check the available Cloudwatch Metrics here.

Notes on Enhanced Monitoring:

• Currently, Physical Devices graphs are not available for Microsoft SQL Server DB instances.

• Currently, viewing OS metrics for a Multi-AZ standby replica is not supported for MariaDB or Microsoft SQL Server DB instances.

Event Notifications

RDS Event Notifications resources include:

• DB instance

• DB snapshot

• DB parameter group

• DB security group

• RDS Proxy

• Custom engine version

RDS Recommendations

Amazon RDS provides automated recommendations for database resources such as DB instances, read replicas, and DB parameter groups. These recommendations offer best practice guidance by analyzing DB instance configuration, usage, and performance data.

Read more here.

AWS Trusted Advisor

From the Trusted Advisor dashboard, you can review the following cost optimization, security, fault tolerance, and performance improvement checks:

• Amazon RDS Idle DB Instances

• Amazon RDS Security Group Access Risk

• Amazon RDS Backups

• Amazon RDS Multi-AZ

Read more here.

CloudWatch Application Insight

When you add your applications to Amazon CloudWatch Application Insights, it scans the resources in the applications and recommends and configures metrics and logs on CloudWatch for application components. Example application components include SQL Server backend databases and Microsoft IIS/Web tiers.

Watch this video for more information.

Read more here.

Others

• Oracle Does not yet support RAC

Conclusion

We have examined various features of popular relational database services (RDBS) including MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server within Amazon's RDS framework. These databases each possess unique strengths and offer different capabilities that can be leveraged depending on the specific requirements of your application. With careful consideration of these features, you can select the RDBS that best fits your needs, thus maximizing performance, cost-efficiency, and overall productivity.

Remember that, as cloud technologies and AWS services continue to evolve, the features and capabilities of these RDBS could change over time. Hence, it is crucial to keep up-to-date with the latest AWS documentation, updates, and best practice guidelines to ensure the optimal use of the selected RDBS.

Disclaimer

This document is intended as a general guide to the features of relational database services within AWS RDS as of the time of writing in July 2023. While every effort has been made to ensure accuracy, the rapidly evolving nature of cloud services means that some information may become outdated or inaccurate over time.

The document does not constitute professional advice, and decisions should not be made solely based on this content. It is recommended that readers refer to the latest official AWS documentation, or consult with an AWS certified professional or the AWS support team for the most accurate and up-to-date information.

Feel free to correct me if I made any mistakes :)

About Web Scraping

Web scraping has been developing in past years. We also have tools that don't require coding to scrape (ParseHub, Octoparse). While it is easy, it lacks flexibility.

Building our scraper from zero allows us to use it however we want. For Python, Scraping can be done using scraping libraries (Requests, BeautifulSoup, Cheerio) and frameworks like Scrapy and Selenium.

In this experiment, we will use BeautifulSoup to parse HTML documents to find the part we want. GCP's AutoML will also be used to extract text from images.

Overview Process

Before we go hands-on with implementation, I want to explain how it will work from accessing the web page until we get the pieces of information that we want.

First, we need to access a web page from which the image we want to extract. In this article, we will use my blog page for privacy reasons, and I want to avoid getting in trouble for scraping another website and publishing an article about it. We aim to get all titles from the article's cover images I post.

After we got the HTML document of the web page, we extracted the image by finding the "img" tag. We should have a list of "img" tags before we move to the next step.

From that "img" tags list we download images by extracting URL from "src" attribute. Download, as binary and store it as variable.

Next, we send a request containing the binary of that images to GCP's Auto ML Vision. There is a free tier on it, don't worry about spending money on this experiment.

The response from AutoML Vision will be a string, but it could be more pretty. We need to make those strings readable and useable for the final part.

Implement

Now that you have a high level of what we will build, let us start coding!

First thing first, we will need to set up the environment. I use venv to keep my computer clean, but it is up to you. There are many ways to do this that I will not explain. Make sure to install packages. Below are the packages that we need.

from io import BytesIO
from bs4 import BeautifulSoup as bs
from google.cloud import vision
import urllib

import re
import requests

Our first method will be to get an HTML document from a URL (my blog page) and extract "img" tags from it. After that, we used links from the "src" attribute, downloaded them, and stored them in a variable. The return value is a list of image files in binaries.

def get_image_files_from_link(url):
    r = requests.get(url)
    soup = bs(r.content)
    image_tags = soup.find_all(
        'img', {'src': re.compile('(.*)(jpg|png|gif|JPG|PNG|GIF)')})
    image_srcs = list(map(lambda x: x.attrs.get('src'), image_tags))

    image_files = []
    for src in image_srcs:
        try:
            src = src.split('=')[1]
            src = urllib.parse.unquote(src)
            response = requests.get(src)
            image_files.append(BytesIO(response.content))
        except Exception as e:
            continue

    return image_files

Using the first method's return value, we create a method to loop it and detect (as in extracting text from an image) and print it. We will implement the "detect" method in a moment.

def get_text_from_image(url):
    image_files = get_image_files_from_link(url)
    titles = []
    for image_file in image_files:
        title = detect(image_file)
        title = parse_text(title)
        if title != '':
            titles.append(title)

    print(titles)

For the "detect" method, we initialize GCP's client, build a request, and send it to GCP. The response will be an object with metadata on how it was processed, but the text_annotations content is essential. It has extracted text from an image.

def detect(image_file):
    try:
        client = vision.ImageAnnotatorClient()

        content = image_file.read()
        image = vision.Image(content=content)

        response = client.document_text_detection(image=image)
        labels = response.text_annotations

        # Get longest text in labels.description
        text = ''
        for label in labels:
            if len(label.description) > len(text):
                text = label.description

        return text
    except Exception:
        return ''

If we return it as it is, we will receive a string with many unnecessary values like space, line breaks, or words we don't want. Here is the example of it.

['aend', 'aend', 'My First Talk\nas an Engineer\nAlvin Endratno\nFull Stack Engineer\nZel', '', 'Many Ways to\nDeploy Docker in\nAWS\nAlvin Endratno\nFull Stack Engineer\nZel', '', '@alvinend\nUsing SQL to Query Data with\nDelta Lake\nAPACHE\nSpark A\nDELTA LAKE', '', '', '@alvinend\nSetup Jupyter in EC2 and\nApache Spark with Delta Lake\nconnection to S3\nAPACHE\nADELTA LAKE', "@alvinend\nConfigure Lambda's Provisioned\nConcurrency in Multi Environments\nAWS SAM\naws", '@alvinend\nComparing React JS and\nSolid JS in Syntax', '@alvinend\nServerless Cron Jobs with\nAWS Batch\naws', '@alvinend\nBuilding AWS EC2 Manager with\nLambda and Slack\naws\n●入米', '@alvinend\nAWS S3 as a Database\nwith S3 Select\naws\n2', '@alvinend\nBuilding Serverless Task Manager:\nIntroduction & Design (Part 1)\nXd', '@alvinend\nKnowing How Javascript\nExecute Its Codes\nUS', '@alvinend\nBuilding CLI Tic Tac Toe Game\nwith Node JS\nUS\nXXO\nOox\nXXX']

To avoid that, we need to parse it. Usually, it is done by writing a Regular Expression (REGEX) or some machine learning, but in this example, we will remove a word that has at least five characters, remove the author's name and trim it.

def parse_text(text):
    parsed_text = ''
    for c in text.split('\n'):
        if len(c) < 5:
            continue

        parsed_text += c + ' '

    parsed_text = parsed_text.replace('@alvinend', '')
    parsed_text = parsed_text.replace('Alvin Endratno', '')
    parsed_text = parsed_text.replace('Full Stack Engineer', '')
    parsed_text = parsed_text.strip()

    return parsed_text

The last thing to do is run it. python3 main.py

if __name__ == '__main__':
    res = get_text_from_image(
        'https://blog.alvinend.tech/')

And the result is below.

['My First Talk as an Engineer', 'Many Ways to Deploy Docker in', 'Using SQL to Query Data with Delta Lake APACHE Spark A DELTA LAKE', 'Setup Jupyter in EC2 and Apache Spark with Delta Lake connection to S3 APACHE ADELTA LAKE', "Configure Lambda's Provisioned Concurrency in Multi Environments AWS SAM", 'Comparing React JS and Solid JS in Syntax', 'Serverless Cron Jobs with AWS Batch', 'Building AWS EC2 Manager with Lambda and Slack', 'AWS S3 as a Database with S3 Select', 'Building Serverless Task Manager: Introduction & Design (Part 1)', 'Knowing How Javascript Execute Its Codes', 'Building CLI Tic Tac Toe Game with Node JS']

It could be better, but it shows how to extract information from images with GCP AutoML Vision.

Closing

In real-life projects, we usually extract information from images to get information unavailable in texts on websites. For example, a company website with a phone number in an image. We can get it by using a regex that determines the phone number in the raw string GCP AutoML provided.

Also, it is crucial to ensure it is legal before scaping websites. Happy Scraping!

Preparation

After agreeing to speak, I was given some time to prepare it. I began building presentation slides. It was a tech event, so I talked about my project, where I was responsible for architecting servers with AWS. We used a rather unpopular service of AWS and managed to create a great outcome with it, making me think that people would like to hear it.

Done with the slides, I asked them to review my slides. My manager, an SRE engineer, and I gathered in a meeting and heard my presentation. I got a lot of feedback on my way of presenting and errors in technical content. I'm glad to have colleagues that have my back.

Finally, we asked the design team to decorate my slides. It was mind-blowing. Here is the first page. It says, "Implementation of Processing Large Amounts of Data using Serverless".

Mainly talked about leveraging AWS serverless services (Lambda, API Gateway, and AWS Batch) to make high-availability and low-cost services. It was similar to my blog post that I published this May.

Event Day

The event was held in Osaka, Japan. At that time, I was in Indonesia, so all the speakers were onsite except me, who talked through video call.

I woke up in the morning, brushed my teeth, ate my breakfast, and rehearsed it many times. From morning until it was my turn, late evening at 9 PM. I was nervous.

Not only is this my first presentation as an engineer, but other keynote speakers are also amazing people. The Head of Product Development, CTO, and Co-Founder speak at that event. The cherry on top, when it came to my turn to speak, there was a technical problem, and my turn was pushed to last. Again I was nervous.

However, a few minutes before my turn, I calmed. I don't know why, but my heartbeats are slowed, my mind is clear, and I can normally speak. The presentation went well.

Several days later, I received an email with comments about my presentation. It was nice to know that people took an interest in what you said.

Closing

As an introverted engineer, I know how hard it is to talk to many peoples. But an opportunity like this won't come forever, and I want to seize every opportunity to the best of my ability.

When you search in google on how to deploy docker in AWS, chances are you will find AWS ECS (Elastic Container Service). In my opinion, it is a good service, if you don't mind the complexity. However, if you are starting and want to deploy it, there may be another way that you will find fit.

Simple compute service

While some people may say that using a server to deploy docker is outdated. It is the simplest way to get your application started and running. If you want to know more about AWS and its concept, I suggest you use EC2 (Elastic Container). But if your only interest is getting stuff running, there is a service called AWS Lightsail.

Platform as a service with AWS Elastic Beanstalk

AWS Elastic Beanstalk helps developers focus on applications by offering a way to quickly deploy and manage applications in the AWS Cloud without learning about the infrastructure that runs those applications. Elastic Beanstalk reduces management complexity without restricting choice or control. You upload your application, and Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.

For deploying docker, AWS Elastic Beanstalk offers two types of platform branches. One is called docker, Running on 64bit Amazon Linux 2, and the other is called ECS, Running on 64bit Amazon Linux 2. If you want to get your application started but with some AWS infrastructure behind it, use Docker Running on 64bit Amazon Linux 2. It will create EC2 for you. On the contrary, choose the latter if you want to build it on ECS so it can be scalable.

The Famous Elastic Container Service

ECS is renowned for some reasons. For many things that ECS does, like managing and orchestrating many containers, health checks, and multi-availability zone clusters, ECS is straightforward and easy to understand, in my opinion. Not to mention it work great with a serverless container service named Fargate. That is if you know AWS.

To build an ECS cluster, you will need to understand how to make EC2 and set up a network in VPC so that ECS can communicate with its agent. Also, you don't want a security issue, so you set up a firewall called a security group. Because you are dealing with a cluster, you will need a load balancer. Besides that, there is also Code Pipeline to build CI/CD and Elastic Cloud Registry (ECR) to store your image before it can be used by ECS.

ECS is an excellent service if you know what you are doing.

Kubernetes with Elastic Kubernetes Service

I only can recommend EKS if you are both comfortable with EKS and Kubernetes. While I said that ECS is a simple service if you know AWS, EKS is, in my opinion, complex even if you know AWS. Learning Kubernetes alone may take some time. The merit that I can think of if that Kubernetes is open source which many cloud providers offer. It gives us the flexibility to change, and because it is open source and widely known, it has a broader appeal for hiring.

Closing

There are many ways to deploy docker, and I think it is caused by the broad need to deploy docker. For instance, someone may want to deploy it without much thinking, but others may need more control over the infrastructure. I am trying to say there are no wrong choices or services.

About Dataset

For this experiment, we will use a dataset about courses, students, and their interactions with Virtual Learning Environment (VLE) for seven selected courses (called modules). You can get it here.

It has enormous data. One of them has around 450 MB of CSV. We will use that to see how fast Delta Lake can insert and query the data.

Import Dataset

After we downloaded the dataset and uploaded it into the server from jupyter, we needed to read it with Apache Spark, write it as delta's parquet, and upload it to S3.

Below is a method to do that.

def import_csv(filename):
    # Read CSV from local directory
    df = spark \
        .read \
        .option("header","true") \
        .csv(f"./dataset/{filename}.csv")

    # Write to S3
    df.write\
        .mode("overwrite")\
        .format("delta")\
        .save(f"s3://s3-bucket-name/table/{filename}/")

After that, call it with a parameter of the name of the CSV name. Below is an example of importing assessment.csv to delta's parquet in S3.

import_csv("vle")

After it succeeds, we can confirm it by going to S3.

Repeat it so that all CSV can be imported. Here is a summary of the file name and execution time.

Querying Data

After we import all our datasets into S3, we can start querying them with SQL. Before that, let me declare one method that I will explain later.

def table_dir(tablename, with_as=True):
    if with_as:
        return f"delta.`s3://s3-bucket-name/table/{tablename}/` AS {tablename}"
    else:
        return f"delta.`s3://s3-bucket-name/table/{tablename}/`"

When querying with delta lake, we need to specify where the folder that holds delta's parquet is. Rather than writing it every time, I made a method of it.

SELECT

Start with an easy one. Let us write a SELECT query.

spark.sql(f"""
    SELECT code_module, count(*) as module_count
    FROM {table_dir('studentVle')}
    GROUP BY code_module
""").show()

Here we query a 400MB table and group it. It only took 1.74 seconds!

JOIN

Let's take it a little further. Try joining three tables.

spark.sql(f"""
    SELECT activity_type, SUM(sum_click)
    FROM {table_dir('studentVle')} 
    INNER JOIN {table_dir('vle')}
    ON studentVle.id_site = vle.id_site
    INNER JOIN {table_dir('studentInfo')}
    ON studentVle.id_student = studentInfo.id_student
    WHERE studentInfo.final_result = "Pass"
    GROUP BY vle.activity_type
""").show()

Here we got two joins, one where and one grouping. Execute time is a little longer at 6.88 seconds, but I got the jobs done.

DELETE

Enough with the read. How about writes? Okay, let's try delete first.

spark.sql(f"""
    DELETE FROM
        {table_dir('assessments')}
    WHERE
        code_module = "AAA"
        AND code_presentation = "2013J"
        AND id_assessment = 1752 
""").show()

And yes, we can delete stuff.

INSERT

Insert it back.

spark.sql(f"""
    INSERT INTO
        {table_dir('assessments', False)}
    VALUES
        ("AAA","2013J","1752","TMA","19","10")
""").show()

We also can insert stuff.

Closing

With this experiment, we know we can use Delta Lake to import and query data with our beloved SQL. It is good to note that what we do today can be done with normal Apache Spark. Next time, we will do an experiment covering Delta Lake features that made it special. Thank you for reading!

Launching EC2

We will not do an EC2 launch tutorial, so I am just going to write simple steps and hope you can manage! (Tips: There are a lot of tutorials about EC2, google it)

1. Log in to your AWS Console
2. Launch EC2 with these setting

That is all! Try to ssh into EC2 before continuing to the next step.

Install Required Tools

Next, let's install python, pip, and pyspark

Install Python

Ubuntu 22.04 LTS ships with the latest toolchains for Python, Rust, Ruby, Go, PHP and Perl, and users get first access to the latest updates for essential libraries and packages. Just in case, let's upgrade packages.

sudo apt update
sudo apt -y upgrade

And check out the python version.

python3 -V

When I wrote this article, my latest python version was 3.10.4.

Install PIP

Although our python is built-in into Ubuntu, it does not come with package manager pip. So let us install it!

sudo apt install -y python3-pip

Same with python, it is always good to check if it is correctly installed or not. Run the below command to check its version.

pip -V

Here is my output.

pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

Install Apache Spark

Running Delta Lake require Apache Spark, so let's install it!

pip install pyspark==3.3

Install Java

Execute the following command to install the JRE and JDK from OpenJDK 11.

sudo apt install default-jre
sudo apt install default-jdk

It is always good to check if it is correctly installed or not. Run the below command to check its version.

java -version

Here is my output.

openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)

Setting up Jupyter

Running your Delta Lake in CLI is cool, but it isn't enjoyable. So, let us install Jupyter in Ubuntu which is accessible from our local computer.

Install Jupyter

Believe it or not, to run jupyter, we need to install jupyter. Here is a command to install it.

pip install notebook

Easy right? But in my case, I had some problems. Although it was installed successfully, I can't run the jupyter command. When I look back at the install log, I found some warnings.

WARNING: The script jupyter-execute is installed in '/home/ubuntu/.local/bin', which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

So I suspect it will work when I add that directory to the path. For adding a path to ~/.bashrc, run the command below.

echo "export PATH=$PATH:$HOME/.local/bin"  | tee -a ~/.bashrc

And I successfully ran jupyter in my terminal.

Expose Jupyter to Public

The next step is to make jupyter accessible from our local computer. First, generate jupyter's config file by executing the command below.

jupyter notebook --generate-config

It should output your config directory. Mine was /home/ubuntu/.jupyter/jupyter_notebook_config.py. Open that file.

vi /home/ubuntu/.jupyter/jupyter_notebook_config.py

Find, Un-comment, and edit these options to expose your juypter

First one is c.Notebook.App.ip. It is to specify the IP address the notebook server will listen on so that we can access it with our EC2 public IP address.

c.NotebookApp.ip = '*'

Second is c.NotebookApp.open_browser and specify it to False. We don't want our Ubuntu to open the notebook when we start the server

c.NotebookApp.open_browser = False

And we are all set. The last thing to do is to start the notebook server.

jupyter notebook

Access it with EC2's public IP, and we should get a notebook similar to this.

Run Spark with Delta Lake

In the final step, let us initiate spark's session and confirm if there are no errors. To do that, we need to create a new notebook in jupyter's console and run the code below. (It is set up to connect with S3)

Note: I tried many ways to make this work, including setting it with the latest version of delta lake and Hadoop (3.3.x), but it throws a java error, and I cannot find a way to fix it. If you can have the latest version working, please let me know in the comments. I referenced the below code from Getting started with Delta Lake & Spark in AWS— The Easy Way post by Irfan Elahi (Thank you!).

from pyspark.sql import SparkSession
spark_jars_packages = "com.amazonaws:aws-java-sdk:1.11.563,org.apache.hadoop:hadoop-aws:3.2.2,io.delta:delta-core_2.12:1.2.1"
spark = (
    SparkSession.builder.master("local[*]")
    .appName("PySparkLocal")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .config("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    .config("spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    .config("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.S3SingleDriverLogStore")
    .config("spark.hadoop.fs.s3a.connection.timeout", "3600000")
    .config("spark.hadoop.fs.s3a.connection.maximum", "1000")
    .config("spark.hadoop.fs.s3a.threads.max", "1000")
    .config("spark.jars.packages", spark_jars_packages)
    .config("spark.sql.sources.partitionOverwriteMode", "dynamic")
    .config("spark.databricks.delta.schema.autoMerge.enabled", "true")
    .config("spark.hadoop.fs.s3a.endpoint", "s3.ap-southeast-2.amazonaws.com")
    .config("spark.hadoop.fs.s3a.aws.credentials.provider", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
    .getOrCreate()
)

And it should install the necessary packages and if there are no errors, congratulations you set up Apache Spark, Delta Lake, and Jupyter in EC2.

Closing

When using Delta Lake for lakehouse in companies, we usually use databricks service, AWS EMR, or services that use for big data processing not traditional servers, maybe that is why there are only a few articles or tutorials that provide a way to deploy it in servers like EC2. Next, I will be performing Delta Lake process in this notebook. Hope this helps you, Cheers!

The Problem

A Lambda function instance created at startup will disappear after a certain period. Therefore, if Lambda is executed after a certain period, there is a good chance that the function instance has disappeared and is re-executed from its creation. This is what we call Cold Start.

Source: https://www.slideshare.net/AmazonWebServices/become-a-serverless-black-belt-optimizing-your-serverless-applications-aws-online-tech-talks/14

If Lambda is re-executed before the function instance is lost, the container is reused, as shown in the following slide, and execution without startup time (warm start) becomes possible. If Lambda functions are executed frequently, the container is more likely to be reused, and the probability of a cold start will decrease.

Solution

My goal is to prevent cold start so that customers using our services will have a good user experience. I could implement an event that fires lambda every minute, but In addition to the time-consuming setup process, There is no fixed amount of time before a function instance goes into an unconscious state. Therefore, periodic execution is only a method to reduce the probability of a cold start and cannot guarantee a warm start.

Another way, which guarantees a warm start and can be done with a bit of setup, is Provisioned Concurrency.

Configuration

In my case, I use AWS SAM Template to Implement it. If you prefer to configure it on the console, I recommend reading AWS Official's blog post about how to do it.

AWS SAM Template Example

It is as simple as adding these three lines to the template.yaml. I will pretend we all know about SAM Template and some basic Code as Infrastructure in AWS.

....

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      .....
      AutoPublishAlias: live
      ProvisionedConcurrencyConfig:
        ProvisionedConcurrentExecutions: 1
      .....

And if you have multiple environments (QA, test, etc.) and want to apply it only on production to save cost. Here is my approach to doing that.

...
Parameters:
    Environment:
    Type: String
    Description: 'Enter Env Name'

Conditions:
  ConfigureConcurrency: !Equals 
    - !Ref Environment
    - prd

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      .....
      AutoPublishAlias: live
      ProvisionedConcurrencyConfig:
        !If 
          - ConfigureConcurrency
          - ProvisionedConcurrentExecutions: 1
          - !Ref "AWS::NoValue"
      .....

It will create an alias but will not set provisioned concurrency in other environments except prd, which is an alias for production.

Note About Provisioned Concurrency

1. Provisioning Concurrency need time, we can set DeploymentPreference (AllAtOnce, Canary10Percent5Minutes, Linear10PercentEvery1Minute etc). as of my research, I couldn't find if it causes downtime or not.
2. Cannot use $LATEST. We must create a version or alias.
3. To make sure we use Provisioned Concurrency, access with arn of the alias. (arn:aws:lambda:REGION:function:FUNTION_NAME:ALIAS_NAME)
4. It is not cheap (sadly). Here is some simple Lambda simulation

As of my finding, here is a way to lower the price:

• Lower Concurrency count. Chances are we don't need it
• Use Application Auto Scaling to scale down outside busy hours.
• Use it only in an environment that needs it.

Closing

Here, I have summarized my understanding. I hope it will be helpful to those who may have difficulty setting it up in AWS SAM template or making it work in multiple environments.

Cheers!

There are many articles regarding how excellent Solid js is and how blazingly efficient it is. But this time, I will focus on syntax. I'm a big fan of React js; react is one framework that gets me a job as Engineer and my first "language" when it comes to frontend framework, so I will be comparing it to react.

State

Let's start with the state. The easiest way to show the state's power is by building a counter. In React, I would be writing it like this.

import React from 'react';

export function App(props) {
  const [count, setCount] = React.useState(0)

  return (
    <div>
      <div>Count: {count}</div>
      <button onClick={() => setCount(count + 1)}>add</button>
    </div>
  )
}

in Solid, it will be

import { render } from "solid-js/web";
import { createSignal } from "solid-js";

function Counter() {
  const [count, setCount] = createSignal(0);

  return (
    <div>
      <div>{count()}</div>
      <button type="button" onClick={() => setCount(count() + 1)}>
        add
      </button>
    </div>
  )
}

Beside Solid called it Signal. It is almost the same. The only difference is when referencing state, Solid needs to call it (count()) while React does not (count).

Props

As for props, it is almost the same. We hand over the value to the child element, and the child element reads it. Because Solid's reactivity occurs on property access on Prop and State objects. Referencing them outside of a binding or reactive computation will not be tracked. Use built-in utilities like mergeProps and splitProps to prevent that.

Effect

Next, I will try to replicate React's useEffect on Solid. Let's say we want to log our count when the state/signal is updated.

import React from 'react';

export function App(props) {
  const [count, setCount] = React.useState(0)

  React.useEffect(
    () => {
      console.log(`Count: ${count}`)
    },
    [count]
  )

  return (
    <div>
      <div>Count: {count}</div>
      <button onClick={() => setCount(count + 1)}>add</button>
    </div>
  )
}

In Solid, I would write it like this.

function Counter() {
  const [count, setCount] = createSignal(0);
  createEffect(() => {
    console.log(`Count: ${count()}`)
  })

  return (
    <div>
      <div>{count()}</div>
      <button type="button" onClick={() => setCount(count() + 1)}>
        add
      </button>
    </div>
  )
}

Pretty similar. With Solid, we don't need to fill out dependencies manually. As for lifecycle, same as React, we can use createEffect without any dependencies (or use onMount), and for the clean up, we can use the onCleanup function. As opposed to React, we use return.

Memo(s)

Unlike React, Solid's components only mount once and don't rerender. Therefore, Solid doesn't need a reference identity function like useCallback. But Solid has a createMemo function for caching purposes. It makes read-only variables that only calculate when dependency (in most cases, prop and signal). In React, we would write it like this.

export function App(props) {
  const [count, setCount] = React.useState(0)

  const calcValue = React.useMemo(
    () => count**100/5**200,
    [count]
  )

  return (
    <div>
      <div>Count: {count}</div>
      <button onClick={() => setCount(count + 1)}>add</button>
      <div>Calc: {calcValue}</div>
    </div>
  )
}

In Solid, we would write it like this.

function Counter() {
  const [count, setCount] = createSignal(0);
  const calcValue = createMemo(() => count()**100/5**200)

  return (
    <div>
      <div>{count()}</div>
      <button type="button" onClick={() => setCount(count() + 1)}>
        add
      </button>
      <div>Calc: {calcValue()}</div>
    </div>
  )
}

Again, the only difference is with Solid, we don't need to declare dependencies, and when referencing it, we call it (. calcValue())

Context

When building a large application, chances are we will need some kind of global state. In React, we have of it; the most popular is Redux. But this time, we will take into built-in function Context. In React, we would write it like this.

const CountContext = React.createContext({
  count: 0,
  setCount: () => void 0
});

export function App(props) {
  const [count, setCount] = React.useState(0)

  return (
    <CountContext.Provider value={{
      count,
      setCount
    }}>
      <Counter />
    </CountContext.Provider>
  )
}

function Counter() {
  const countContext = React.useContext(CountContext)

  return (<>
    <div>Count: {countContext.count}</div>
    <button onClick={() => countContext.setCount(count => count + 1)}>add</button>
  </>)
}

In Solid, we could write like this.

const CountContext = createContext({
  count: 0,
  setCount: () => void 0
})

function App() {
  const [count, setCount] = createSignal(0)
  const store = {
    count,
    setCount
  }

  return (
    <CountContext.Provider value={store}>
      <Counter />
    </CountContext.Provider>
  );
}

function Counter() {
  const countContext = useContext(CountContext)

  return (<>
    <div>Count: {countContext.count()}</div>
    <button onClick={() => countContext.setCount(count => count + 1)}>add</button>
  </>)
}

There is not much to comment on here; we can write it the same way.

Ending

I'm surprised at how similar React and Solid are and excited about Solid in the future. What do you think

What is AWS Batch

AWS Batch is a set of batch management capabilities that enables developers, scientists, and engineers to quickly and efficiently run hundreds of thousands of batch computing jobs on AWS.

AWS Batch can be integrated with serverless container service (AWS Fargate) and set so that we only pay for what we use.

Build Batch

First of all, we need to build a working batch job.

Write the code

The easiest way to build a batch, in my opinion, is by creating a docker image. As long as it runs on a docker container, you can run any language in AWS Batch. In this example, we will be using python.

Dockerfile

To create a docker image, we need Dockerfile. We will create a simple python container and install the package to our requirements.txt.

./Dockerfile

FROM python:3.9

ADD . /app

WORKDIR /app

RUN pip install -r requirements.txt

VOLUME /app

Sample Code

For this example, we will run the downloaded package and test it if we can confirm it in the execution logs.

requirements.txt

requests==2.27.1

run.py

import requests

res = requests.get("https://blog.alvinend.tech/")
print(res.text)

Put Image in ECR

The next step is putting our image in ECR.

Create New Repository

1. Go to AWS ECR (Don't forget to check your AWS region)
2. From the sidebar, go to "Repositories" and click the "Create repository" button.
3. Choose "Visibility Settings" to "Private," Enter the repository name.

Push Image

Select the created repository and click the "View push Command" button.

Run those four commands and check if the repository is updated from the AWS console.

P.S. If you face any error while pushing the image, there is a good chance there are authentication problems. The link below might help you.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Execute Batch in AWS Batch

There are four steps before we can run our job.

Create Compute Environment

As the name suggests, Compute Environment is where our jobs will be executed. There are two types of Compute environments.

• EC2
• EC2 Spot
• Fargate
• Fargate Spot

If you choose EC2, AWS will start a new EC2 Server and run your job there. Fargate is an AWS Serverless container service. Spot is an option to use surplus resources for a low price in exchange for the risk that the resource might be suddenly unavailable. We will be using Fargate Spot.

• Go to AWS Batch Page
• Click "Compute Environment" from the sidebar
• Click the "Create" button
• Fill out Create Form
  • For "Compute environment configuration."

• For "Instance configuration."

• For "Networking." Just leave it! Default VPC has a public subnet that can connect internet.

• Click "Create."

Just like that, we created our computing environment.

Create Job Queue

Jobs are submitted to a job queue where they reside until they can be scheduled to run in a Compute Environment.

• Go to AWS Batch Page
• Click "Job queues" from the sidebar
• Click the "Create" button
• Fill out Create Form
  • For "Job queue configuration."

• For "Connected compute environments," select compute environment that we created before.

• Click "Create."

Create Job Definition

Job Definition is a blueprint for creating jobs.

• Go to AWS Batch Page
• Click "Job Definition" from the sidebar
• Click the "Create" button
• Fill out Create Form
  • For "Job type," Choose "Single Node" just because we don't need parallel execution.
  • For "General configuration," Enter any name you want and timeout of 300 (5 minutes).
  • For "Platform compatibility," Enter Fargate with the latest version, Check "Assign Public IP," and execution role to default.
  • For "Job configuration."

And leave the rest to default.

• Click the "Create" button

Create Job

Preparation is done, and let us test-run it.

• Stay on the "Job Definition" Page
• Select created job definition
• Click on the "Submit new job" Button
• Enter job name (again, whatever you want) and choose the job queue that we created
• Click "Submit"

Congratulations ! you have created your first AWS Batch job 🎉. Let's sit back and wait for our execution logs (Hint, it's not real-time, and there are delays)

Logs:

Scheduled Jobs

The last thing we need to do is set a schedule because, without it, this will be a "job," not a "cron job." 😆

Set Rule in AWS Eventbridge

• Go to AWS Eventbridge
• Click the "Create Rule" button
  • For "Define rule detail," Enter "Name" and "Description" with anything you like. Leave "Event Bus" to default and Rule type to "Schedule."
  • For "Define schedule," Select what schedule you want to run the job. See AWS Schedule Expressions
  • For "Select target(s)," Select target types to "AWS Service" and "Select a target" to "Batch job queue." Enter your resource ARN for Job Queue and Job Definition. Enter any name you like in the Job Name.
  • Skip "Configure tags."
  • Review and create

If done correctly, it will appear in the "Rules" list.

Ending

This time we created a serverless job in AWS. Compared to AWS Lambda, AWS Batch need some work to set up, but both have unique use cases, and in my opinion, no tool can fit all scenarios. Knowing many tools on the internet will help us in our journey. Besides, I am happy there are many tools we can use 😃.

Control EC2 Instance with Slack Chat with Lambda

We will start with building a slack bot for controlling our EC2 Instance. For this, we need two lambda functions and an API Gateway. The first function we will name itec2-slack-handler function. It will forward requests from API Gateway that came from the slack bot into our second function, ec2-manager. ec2-manager will stop and start function for us.

Build Slack Handler

You might be thinking, why do we need a separate lambda function for handling slack event subscription, which will send a request to API Gateway when there are chats in slack and lambda function for a start or stopping EC2 Instance. I tried to do it in one function. After I finished implementing it, I was encountered a strange bug. When I trigger EC2 Instance one time, I get up to four triggers in my lambda. After many hours of research, I found out that you must return a response to slack. So, to avoid this bug, we need to make two lambda functions.

Your app should respond to the event request with an HTTP 2xx within three seconds. > If it does not, we'll consider the event delivery attempt failed. After a failure, we'll retry > three times, backing off exponentially.

Maintain a response success rate of at least 5% of events per 60 minutes to prevent automatic disabling.

Respond to events with a HTTP 200 OK as soon as you can. Avoid actually processing > and reacting to events within the same process. Implement a queue to handle inbound events after they are received.

https://api.slack.com/apis/connections/events-api

Slack Handler Function

This time we will be using python in lambda. Here is step by step on setting up the EC2 Instance bot in your slack.

Create Lambda function with python 3.9 runtime.

First of all, we will create a lambda that forwards slack bot requests to the slack manager and return 200 responses as soon as they come.

Before writing our "forward request" code, we need to finish the challenge that slack gives up when subscribing to the event.

import json
import boto3

lambda_client = boto3.client('lambda')

def lambda_handler(event, context):
    print(f"Received event:\n{event}\nWith context:\n{context}")

    slack_body = event.get("body")
    slack_event = json.loads(slack_body)
    challenge_answer = slack_event.get("challenge")

    return {
        'statusCode': 200,
        'body': challenge_answer
    }

Lambda alone cannot connect to slack. We need our entry point, and it is API Gateway. AWS makes it easy for us to do that.

We are done with setting up our environment in AWS for now. Please take note of our API Gateway Endpoint.

Next step, lets set up our slack apps.

Setting up Slack Apps

We want to make a bot that sends our chat in a certain channel to be sent to API Gateway. Here is how we make Slack Bot:

1. Go to https://api.slack.com/ and click on the Your Apps Button
2. Login and create a new app in your workspace

After you create your app, set Event Subscription to Lambda

1. Go to Event Subscription by accessing it from the sidebar.
2. In Request URL input, enter API Gateway Endpoint that we memo earlier.
3. In "Subscribe to bot events" add "message.channels" user event.
4. Save changes.

What we set until now is a flow for our chat in slack to be sent to API Gateway as a request. Besides that, we need to develop a flow for our Lambda function to be able to send chats to our channel in slack.

1. From the sidebar, go to "OAuth & Permissions".
2. In the "Scopes" section, set "Bot Token Scopes" to be the same as below.

Press install to workspace, and don't forget to memo bot token that begins with xoab-.

Configure Slack Handler to forward Function

Since we finished with our integration setting, we can delete our current code and replace it with the "request forwarding" code.

import json
import boto3

lambda_client = boto3.client('lambda')

def lambda_handler(event, context):
    print(f"Received event:\n{event}\nWith context:\n{context}")

    try:
      lambda_client.invoke(
        FunctionName='ec2-manager',
        InvocationType='Event',
        Payload= json.dumps(event)
      ) 

    except Exception as e:
      print(e)

    return {
      'statusCode': 200
    }

Setting up EC2 Manager Lambda Function

Halfway there! We just need to create a lambda function two manage our instance, in this case, starting and stopping our EC2 Instance.

First thing first, we need to create and code it.

import boto3
import os
import urllib.request
import json

ec2 = boto3.resource('ec2')
INSTANCE_ID = 'YOUR_INSTANCE_ID_HERE'

def send_text_response(event, response_text):
    channel = event.get("channel")
    ts = event.get("ts")
    print("Messaging Slack...")
    SLACK_URL = "https://slack.com/api/chat.postMessage"

    data = urllib.parse.urlencode(
      (
        ("token", os.environ["BOT_TOKEN"]),
        ("channel", channel),
        ("thread_ts", ts),
        ("text", response_text)
      )
    )
    data = data.encode("ascii")

    request = urllib.request.Request(SLACK_URL, data=data, method="POST")
    request.add_header( "Content-Type", "application/x-www-form-urlencoded" )

    print('Fire off the request!')
    x = urllib.request.urlopen(request).read()
    print(x)

def start_workspace():
    instance = ec2.Instance(INSTANCE_ID)
    if instance.state.get('Name') == 'running':
          return f"Instance Already Running."

    # Start EC2 Instance
    instance.start()

    # Wait Until its Run
    while True:
      instance = ec2.Instance(INSTANCE_ID)
      if instance.state.get('Name') == 'running' and instance.public_ip_address is not None:
        return f"Instance Started Successfully. \n Public IP: {instance.public_ip_address}"    

def stop_workspace():
    instance = ec2.Instance(INSTANCE_ID)

    if not instance.state.get('Name') == 'running':
        return f"Instance Already Stopped."

    # Stop EC2 Instance
    instance.stop()

    # Check Until is Stopping
    while True:
      instance = ec2.Instance(INSTANCE_ID)
      if instance.state.get('Name') == 'stopped' or instance.state.get('Name') == 'stopping':
        return f"Instance Stopped Successfully."

def lambda_handler(event, context):
    print(f"Received event:\n{event}\nWith context:\n{context}")
    slack_body = event.get('body')
    print(f"Body:\n{slack_body}")
    slack_event = json.loads(slack_body).get('event')
    user_id = slack_event.get("user")
    text = slack_event.get("text")

    try:
      if text != None and ('Start Instance' in text):
        print("Starting Instance")
        response_text = start_workspace()
        if response_text:
          send_text_response(slack_event, response_text)

      if text != None and ('Stop Instance' in text):
        print("Stopping Instance")
        response_text = stop_workspace()
        if response_text:
          send_text_response(slack_event, response_text)

    except Exception as e:
      print(e)

    return {
      'statusCode': 200
    }

So this is how our code works:

1. When there is a "Start Instance" keyword in chat, it will trigger EC2 start and reply public IP when the process is a success
2. When there is a "Stop Instance" keyword in chat, it will trigger EC2 start and reply public IP when the process is a success

But, for our lambda to be able to start or stop our instance, we need to permit it to do so.

• Go to the Configure tab and click Role name

• Add a needed policy to start and stop the instance. if you don't care about security and stuff, attach the EC2FullAccess Policy

And for our lambda to be able to send messages through Slack API, we need an access token, which is given through the environment variable.

• Go to the Configure tab and click Role name
• Add BOT_TOKEN with the value of Slack Bot Token that we memo earlier

Testing

If we set everything upright, we should have our bot happy and running.

Starting up Instance:

Stopping Instance:

Closing

It was a long journey. We went through AWS, Slack, AWS, and back to Slack to trigger AWS. Although we built an EC2 instance manager this time, there are many ways to use this flow. Like they always say, the sky is the limit, Happy Hacking!

Prerequisite

In this experiment, We will be using AWS S3 and Python. Here is what I have prepared before the experiment.

• AWS Account
• S3 Bucket to store CSV File
• Python Environment
• A Dummy CSV File filled with one thousand users' information

Implementing CRUD

And now, let's get our hands dirty and code this thing. First, of Its a good thing to know what we are going to build, so I wrote a test (not really) before implementing it.

from s3.user import read, delete, update, create

# Read Query
print("Read Data")
print(read("WHERE firstname = 'Devina'"))

# Output
# [['1274', 'Devina', 'Terencio', 'Devina.Terencio@yopmail.com', 'Devina.Terencio@gmail.com', 'doctor']]

# Update Query
print("Update Data")
update(
  "WHERE firstname = 'Devina'",
  {
    'lastname': 'Green'
  }
)

# Output
print(read("WHERE firstname = 'Devina'"))
# [['1274', 'Devina', 'Green', 'Devina.Terencio@yopmail.com', 'Devina.Terencio@gmail.com', 'doctor']]

# Delete Item
# print("Delete Item")
delete("WHERE firstname = 'Devina'")

# Output
print(read("WHERE firstname = 'Devina'"))
# []

# Create Item
print("Create Item")
create({
  'firstname': 'Devina',
  'lastname': 'Terencio',
  'email': 'Devina.Terencio@yopmail.com',
  'email2': 'Devina.Terencio@gmail.com',
  'profession': 'doctor'
})

# Output
print(read("WHERE firstname = 'Devina'"))
# [['2100', 'Devina', 'Terencio', 'Devina.Terencio@yopmail.com', 'Devina.Terencio@gmail.com', 'doctor']]

If what is printed in the terminal is the same as expected, we successfully built our application.

Here is the GitHub link: https://github.com/alvinend/s3-as-a-db

Read

What do we do when we want to query some items.

• Use S3 Select to query CSV. This will return object
• Get payload string from the response object
• Change payload string to list

Easy!

import boto3
import pandas
import settings

s3_client = boto3.client('s3')

def read(
    query,
    select='*'
):
    # Use S3 Select to Query CSV
    res = s3_client.select_object_content(
        Bucket = settings.BUCKET_NAME,
        Key = 'user.csv',
        ExpressionType = 'SQL',
        Expression =f"Select {select} from S3Object s " + query,
        InputSerialization = {
            'CompressionType': 'NONE',
            'CSV' : {
                'FileHeaderInfo' : 'Use',
                'RecordDelimiter' : '\n',
                'FieldDelimiter' : ','
            }
        },
        OutputSerialization = {
            'CSV' : {
                'RecordDelimiter' : '\n',
                'FieldDelimiter' : ','
            }
        }
    )

    records = ''

    for event in res['Payload']:
        if 'Records' in event:
            records = event['Records']['Payload'].decode('utf-8')

    # Change String to Array 1, "Name1, Address1\n 2, Name2, Address2\n" -> ["1, Name1, Address1", "2, Name2, Address2", ""]
    records = records.split('\n')

    # Remove Empty String Element ["1, Name1, Address1", "2, Name2, Address2", ""] -> ["1, Name1, Address1", "2, Name2, Address2"]
    records = filter(len, records)

    # Change Elemnt to be String [["1", "Name1", "Address1"], ["2", "Name2", "Address2"]]
    records = list(map(lambda x: x.replace('\r', '').split(','), records))

    return records

Delete

Since S3 has no feature to manipulate CSV files, we must do it manually. By manually, I mean taking the file from S3 manipulate and replacing the old file with the manipulated file.

import boto3
import pandas
import os
import settings
from io import StringIO

s3_resource = boto3.resource('s3')
s3_client = boto3.client('s3')

def get_df():
    res = s3_client.get_object(Bucket=settings.BUCKET_NAME, Key="user.csv")

    df = pandas.read_csv(res.get("Body"))

    return df

def save_to_s3(df):
    csv_buffer = StringIO()
    df.to_csv(csv_buffer, index=False)
    s3_resource.Object(settings.BUCKET_NAME, 'user.csv').put(Body=csv_buffer.getvalue())

We will use this method to create and update operations too. For delete, we do these actions.

1. Get Dataframe from S3
2. Get Ids from S3 Select query
3. Filter out selected ID
4. Save Dataframe to S3 as CSV

from s3.user.utils import get_df, save_to_s3
from s3.user.read import read

def delete(query):
    # Get Dataframe
    df = get_df()

    # Get Ids from query
    ids = read(query, select='id')
    ids = list(map(lambda x: int("".join(x)), ids))

    # Filter out Selected ID
    newdf = df[~df.id.isin(ids)]

    # Save Dataframe to S3 as CSV
    save_to_s3(newdf)

Update

The update operation is almost the same as delete, with the exception instead of filtering it, we update it.

1. Get Dataframe from S3
2. Get Ids from S3 Select query
3. Update selected ID with new values.
4. Save Dataframe to S3 as CSV

def update(
    query,
    data_dict
):
    update_column_key = list(data_dict.keys())
    update_column_value = list(data_dict.values())

    # Get Dataframe
    df = get_df()

    # Get Ids from query
    ids = read(query, select='id')
    ids = list(map(lambda x: int("".join(x)), ids))

    # Update DF
    df.loc[df.id.isin(ids), update_column_key] = update_column_value

    # Save Dataframe to S3 as CSV
    save_to_s3(df)

Create

Last but not least is the create operation. Although I would like to insert data in pandas' DataFrame, it has no auto-increment feature. So we have to manually search the latest stored ID, increment it, and save new data.

1. Get Dataframe from S3
2. Get latest id from S3 Select query
3. Create new data with the incremented latest ID
4. Save Dataframe to S3 as CSV

from s3.user.read import read
from s3.user.utils import get_df, save_to_s3

def create(
    data_dict
):
    # Get Dataframe
    df = get_df()

    # Get Ids from query
    max_id = read('', select='MAX(cast(id as int))')
    max_id = int(max_id[0][0])

    # Update DF
    data_dict['id'] = max_id + 1

    df = df.append(data_dict, ignore_index=True)

    # Save Dataframe to S3 as CSV
    save_to_s3(df)

Closing

There you have it, S3 as a database! I know it is slower than a conventional relational database like MySQL and has no ACID compliance. But it is cheap when you are experimenting or making a prototype of something. It is nice to have a choice that does not require you to manage the database and pay monthly server costs.

As always, thank you for reading this post!

What do we need

Before jumping to code, we need to have a clear image of what we will build. To do that, I listed up what this application should be able to do.

• Create, read, update and delete tasks
• Have user authentication
• Can assign tasks to a user
• Can list ongoing tasks that the user has
• Can set a deadline on tasks
• Notify user when a task is assigned
• Remind user when task is over deadline

What should it look like

Again, to have a clear image of our goal, we should have a rough idea of what our application will look like. So making UI design is an excellent start to doing that. We want to keep it simple, easy to use, and dark because everybody loves the dark user interface this day. I jumped to UI Hut for inspiration and started making my very own UI Design with Adobe XD.

What do we use to make it

In my opinion, there is no right or wrong in choosing what tech stack when building something. What I think is important is why do we use it. Here is our's task stack

• AWS API Gateway & Lambda & DynamoDB for the backend. We work for a small number of people and do not plan on accessing it frequently. It will be a waste to deploy a whole server just for it. Serverless may be expensive in some use cases, but for low in-frequent access, it is perfect.

• React for the frontend. We have our REST API in serverless, and I think it makes sense to use SPA for our application. The reason why do we use React is because I used to use React, and I love to react. After building it, we will deploy it in S3 together with Cloudfront.

• React Native for the mobile when tasks are assigned or have a task over deadline, we want to be notified. While I think it is overkill to build a whole mobile application for it, it is an excellent chance to try new technology and have it with it.

Ending

I will be building this application in the next two or three months. I am excited to show how I create the application and how I think what and how it should be done.

Thank you for reading this post!

Javascript is a single-threaded language

Do you know that our brain is not able to do multitask? Except for a tiny percentage of people, When we think we’re multitasking, most often, we aren’t doing two things at once. Still, instead, we’re doing individual actions in rapid succession or task-switching. Single-threaded language like Javascript does the same thing. It stacks up a list of tasks and executes them one by one. That stack has a name, and it is "Call Stack".

From the top, code is inserted per task, and if a task is a function and has another task in it, it will push into the stack. Once one set of processes is pushed in, Call Stack will consume it in a LIFO manner. The last task pushed will be the first one to be consumed.

When one set of processes in this case function is finished, Javascript Runtime will continue to execute the rest of the codes.

How About Promises?

If you know Javascript and build some kind of application, 9 out of 10, you already know about Promise.

What is Promise

Maybe you already wrote it but didn't know that your code is Promise, so let us start with an example.

async function getUser() {
  const res = await axios.get('/user')

  return res.body
}

HTTP Request is a common thing we do when we build Web Apps. In that example, we get user data from the backend end to return its content.

Let us test that function.

const user = getUser()
console.log(user)

We expect it to log out user data. But here, the output says otherwise.

undefined

It happens because we don't until our data is fetched. To fix it, we insert await at declaration.

const user = await getUser()
console.log(user)

And the output is as we are expected.

{
  name: 'Alvin'
}

Promise in Javascript is an object that may produce value in the future. Since JavaScript is an asynchronous language, the following process will be executed without waiting to complete the previous one if the previous execution takes a long time. But if we wait for Promise to end with await, we get our expected value.

The question here is, "How does javascript do something asynchronously while itself is single-threaded?".

Javascript concurrent-like process

I know it's a little bit misleading, but while javascript runtime can only do one thing simultaneously, the browser does not only consist of javascript runtime.

WebApis

To make it simple, WebApis is where asynchronous code is executed. When code contains an asynchronous function, the code is then pushed to Call stack. Call Stack know it is an asynchronous task and throw it to WebApis. WebApis and then process it and throws callback into Callback Queue.

WebApis also contain DOM and also handles user interactions. For example, when the user clicks a button, WebApis throws a click callback into Callback Queue.

Callback Queue

Callback Queue is a queue of tasks thrown by WebApis. Tasks will later give back to Call Stackand be processed.

Event Loop

The process of moving tasks from main code to call stack, callback queue back to call stack is the event loop. Event Loop stops when the call stack processes something and loop when there are no tasks left. Javascript will execute all main code processes in that logic before touching Callback Queue.

Executing

Let's get a better image of it. We start with simple code, the same as before, rather than only logging strings. We try to make an async request, fetch a user, and log out the user name.

Event Loop will start its loop at the call stack and process the main code as before. We have an async function, so instead of processing in the call stack, it will be thrown to WebApis to be executed there.

It will continue until codes and stacks are processed. When WebApis is done executing, it will throw a callback in the Callback queue. In this case, it's logging out the user name.

After that, Event Loop will start looping between Callback Queue and Call Stack. If there are tasks in Callback Queue, It will throw it to Call Stack and execute it later when the event loop stops at Call Stack.

Ending

I hope you get the big picture of how javascript runs in our browser. If you are interested in more, I suggest you watch this video; it's the easiest-to-understand video, in my opinion.

Youtube Link