Pre-commit

A pre-commit git hook is a shell file located inside the git repository in the following location .git/hooks/pre-commit. Infact, when you create a git repository, a sample file is available at .git/hooks/pre-commit.sample. In this blog, we will look at how to use pre commit hook to automatically format and lint python and SQL code.

SQLFluff

sqlfluff is a python base formatter and linter for SQL files that is used in most dbt-sql models. The configuration of the sqlfluff can be achieved by updating the pyproject.toml a configuration format supported by python. The configuration corresponding to sqlfluff for bigquery in dbt is as follows,

[tool.sqlfluff.core]
templater = "jinja"
dialect = "bigquery"
# sql_file_exts = .sql,.sql.j2,.dml,.ddl

[tool.sqlfluff.rules]
comma_style = "leading"
tab_space_size = 4
max_line_length = 80

[tool.sqlfluff.indentation]
indented_joins = false
indented_using_on = true
template_blocks_indent = false

[tool.sqlfluff.templater]
unwrap_wrapped_queries = true

[tool.sqlfluff.templater.jinja]
apply_dbt_builtins = true

Formatting and Linting with SQLFluff

To format sql code before commit, you need to get a list of SQL files that are staged for commit. This script assumes, all your sql files end with *.sql. However, its simple to change the script if you have any other custom file extension.

# Get the list of SQL files staged to be committed
sql_files=$(git diff --cached --name-only --diff-filter=ACM | grep '\.sql$')

Once you get the list of staged files, you can iterate over the list of files and fix them as given below

# fix sql files with sqlfluff
for file in $sql_files
do
    sqlfluff fix  "$file"
done

Upon formatting and fixing SQL files, they need to be staged again for commit

# stage files again with extension .sql
git add -u '*.sql'

Once staged, the sql code needs to be linted again for any errors.

for file in $sql_files
do
    sqlfluff lint  "$file"
    if [ $? -ne 0 ]; then
        echo "sqlfluff found errors in $file. Commit aborted."
        exit 1
    fi
done

So, the full script appears as follows

#!/bin/bash

# Get the list of SQL files staged to be committed
sql_files=$(git diff --cached --name-only --diff-filter=ACM | grep '\.sql$')

# fix sql files with sqlfluff
for file in $sql_files
do
    sqlfluff fix  "$file"
done

# stage modified files again
git add -u

# Lint each sql file staged with sqlfluff
for file in $sql_files
do
    sqlfluff lint  "$file"
    if [ $? -ne 0 ]; then
        echo "sqlfluff found errors in $file. Commit aborted."
        exit 1
    fi
done

Conclusion

sqlfluff is a package that can be used to format and lint sql code for many dialects for dbt. In this blog post, we discuss how to use pre-commit hooks of git to automatically format and lint sql codes.

Passing local environment variables

One of the challenges in using dockers is to pass local environment information to the docker. There are a couple of ways to do this.

1. Using the .env file in the same location as docker-compose. The variables provided in the .env file is used to expand the environment of the docker-compose command
2. The env_file that is provided in the docker-compose.yml file. The variables provided in the env_file are injected into the environment of the containers.

Postgresql Docker Example

To see the difference in action, lets work with an example of a postgresql docker container. The docker-compose.yml file for this is given below.

version: '3.8'

services:
  postgres:
    container_name: postgres_container
    image: postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-admin}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changemenow}
      POSTGRES_DB: ${POSTGRES_DB:-postgres}
      PGDATA: /data/postgres
    volumes:
      - /home/ram/data/docker_data/postgresql:/data/postgres
    env_file: ./config/postgres.env
    ports:
      - "5432:5432"
    networks:
      - postgres
    restart: unless-stopped

networks:
  postgres:
    driver: bridge

Passing username and password variables

In this configuration, the default values are provided for the database configurations. However these values can be over-ridden by passing environment variables. For example, - The databaser user is admin. However this value can be over-riden by passing the variable POSTGRES_USER. Similar over-rides can be done for password and database name.

Lets create two files .env with the same following values

POSTGRES_USER=admin2
POSTGRES_PASSWORD=dontchange
POSTGRES_DB=localpg

Now when we try to verify if the variable values are passed to docker-compose using the command

sudo docker compose convert

we get the following output where the postgres variable values are correctly substituted.

services:
  postgres:
    container_name: postgres_container
    image: postgres
    environment:
      POSTGRES_USER: admin2
      POSTGRES_PASSWORD: dontchange
      POSTGRES_DB: localpg
      PGDATA: /data/postgres
    volumes:
      - /home/ram/data/docker_data/postgresql:/data/postgres
    env_file: ./config/postgres.env
    ports:
      - "5432:5432"
    networks:
      - postgres
    restart: unless-stopped

networks:
  postgres:
    driver: bridge

However, if we create a file ./config/postgres.env with the same values, these variables are not subsituted and we get the following values and we get the default values.

Passing variable to container environment

Lets say we plan to create a custom folder in the container environment (/data/customdir), lets define it in .env file

CDIR=/data/customdir`

Now if we start the container (sudo docker compose up) and try to find the variable in the container environment with following command

sudo docker exec -t postgres_container echo $CDIR

the variable was not defined and we get an empty value returned. However when the same variable (CDIR) is defined through the file ./config/postgres.env, the echo command in docker environment returns the expected value of /data/customdir

Conclusion

To pass a local environment variable to expand/build the docker-compose.yml file, we need to define the variable in the .env file. However, to pass the variable to the container environment, we can define it in any custom file (ex: ./config/postgres.env), but this location needs to be provided under the env_file part of the yml file.

One of the major pain points for me to work with AWS is the time that I need to spend every day trying to provision an EC2 Spot Instance. While, it is much cheaper to use Spot instance, it takes a lot of productive time going through the same routine every day.

While there are full fledged enterprise level IT automation tools like ansible which are more useful to handle systems in production, for individuals working on development and prototyping, this can be easily achieved using boto3 package of python along with crontab in linux to automate this boring step.

Config File

The first step is to create a config file as shown below. Lets name it config.cfg. The file inputs are as follows.

[EC2]
ami=ami-0885b1f6bd170450c
key_pair=keypair_test
max_bid=0.01
instance_count=1
instance_run_type=one-time
instance_machine_type=t2.micro
instance_interrupt_type=terminate
instance_availability_zone=us-east-1f
security_group=sec_group1
iam_role=
subnet_id= 
tag=test-spot-instance
team=me
created_by=me
application=qa
product_description=Linux/UNIX
username=ubuntu

Let us go through the important options here.

1. ami should contain the ami-id from which the instance needs to be provisioned. Here I have used the ami corresponding to Ubuntu 20.01.
2. key_pair is the name of the .pem file (keypair_test.pem) that will be used to tunnel into the virtual instances.
3. max_bid is the maximum value you are willing to pay for the instance. The higher the price, the less likely the spot instance to be terminated.
4. Then you can specify various instance related information such as instance_count, instance_run_type, instance_machine_type and instance_interrupt_type. If required, you can also specify the zone where you need the instance.
5. If you need to attach the instance to a specific security group, you can use the security_group where the value is the name of the security group. Similarly, we can include information of relevant subnet_id and iam_role if needed.
6. The other information such as tag, created_by and application are tag values that are included to the EC2 resource tag.

Python code to create EC2 instance

The code that uses the config file to provision an EC2 spot instance is called ec2_create.py and is provided in this github repository. The repository also has sample config file (config_sample.cfg) that can be customized and used. The code can be run as follows

python ec2_create.py --config config.cfg

This will create the following output

Reading Config File
Spot instance request success, status: open
Waiting... None
Instance allocated ,Id:  <ec2-id>

Now this code can be easily scheduled for daily operation using crontab. This webpage provides an excellent introduction to crontab. I prefer to collect all my crontab jobs into a single folder. So for this, I created a script create_ec2.sh with the following code

/path/to/python /path/to/ec2_create.py --config /path/to/config.cfg

Save the script in a common folder which in my case is ~/crontabJobs. Make the script executable using the following command

chmod +x ~/crontabJobs/create_ec2.sh

To edit crontab, type the following command

crontab -e

This opens the file for edit. If we need to provision the instance everyday of the month at 08:00AM we can simply add the following line

0 8 * * * /home/username/crontabJobs/create_ec2.sh >> /home/username/crontabJobs/cron.log 2>&1

where

1. first 0 indicates minutes,
2. 8 indicates hour of day,
3. the next three stars indicate the day of the month, month of the year and day of week.

If you do not have a local linux machine, you can keep a t2.micro instance running in AWS and perform all these tasks from that instance.

Conclusion

The provisioning of EC2 spot instance that has lower cost can be automated using boto3 and crontab and save sometime from these boring tasks.

1. Help to reduce the time required in processing paper forms.
2. They can digitize information from old important records that can be easily looked up for reference or can be analyzed to build insightful data models.
3. Finally, OCR can be used along with images and videos to extract important textual information.

Tesseract is one of the most important opensource OCR engines available today. At the time of writing this blog, Tesseract can extract text on more than 100 languages. The latest tesseract stable version (4.1.1) encompasses a prebuilt OCR engine built upon long short term memory (LSTM) based neural network models. These models can extract text in many real world datasets. However, in case of images that suffer from poor quality, extracting text with high accuracy requires improving image quality using advanced image processing algorithms.

OpenCV is a great opensource tool that is built on C++ with python bindings that can help you develop and apply various advanced image processing algorithms. In the context of text extraction, some of the important image processing approaches include,

1. Correcting the text orientation (deskewing).
2. Improve the character borders. This can be achieved by converting the image to grayscale and thresholding.
3. Increasing the contrast of the image. This can typically be achieved by histogram equalization that converts a normal distribution of pixel intensities to uniform distribution.
4. Denoising and smoothening of the images using interpolation algorithms.

I have been working over the past few weeks on using OCR to extract text from images. In this article, I will restrict the discussion to denoising the images.

Machine and Software Information

1. Local Machine: Windows 10
2. Tesseract Version: v5.0.0-alpha.20200328
3. Python Version: 3.7.3
4. OpenCV Version: 4.3.0.36

Handling Low Resolution Noisy Images

Consider the following image from which text needs to be extracted

Although, the image section has low resolution (15, 45) and is very noisy, a human eye can clearly see the number present as 166.3. This is possible, because we are able to interpolate the signal between the pixels. However, the prebuilt OCR models in tesseract is unable to perform this interpolation. For example, if we pass this image to pytesseract (the python bindings of tesseract) and use the following standard configs, it does not return any text.

pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"
custom_config = r'-c tessedit_char_whitelist=0123456789. --oem 3 --psm 6 outputbase digits'
print(pytesseract.image_to_string(img_bbox, config=custom_config))

One way to overcome this problem is to denoise and smoothen the image. We can accomplish this by resampling to improve the pixel density. The new pixel values are interpolated from existing signal values.

Mathematical concepts of Interpolation

This section discusses about the mathematical concepts of interpolation. If you are only interested in applications, feel free to skip this section. In most interpolations, to resample at new we apply a discrete convolution operation on the input signal (here image pixels) with a kernel operator.

Where \(a\) is the region around the current signal point that determines the size of the kernel. There are various types of kernels used to perform such interpolations namely (i) box kernel, (ii) Gaussian kernel etc. However, one of the most important and effective kernels in providing a smooth interpolation of images is the Lanczos kernel \(L(x)\) which is defined as follows

The a value is a parameter that is a positive integer which determines the size of the Kernel. Restricting the Kernel size to a fixed constant value ensures that the interpolation operation is an \(O\)(N) operation, where N is the number of signal points to be sampled. The Lanczos kernel is built from the normalized \(sinc(x)\) function whose roots are the non-zero integers as shown below

As you can see from the above image, the Lanczos kernel is smooth, continous and its magnitude dies of rapidly away from the origin. The discrete convolution of this kernel with the original signal (\(f(x)\)) data provides the new resampled/interpolated signal data (\(F(x)\)). In mathematical terms,

In case of multidimensional signal (2D signal as in case of images), the interpolated signal is obtained as pointwise multiplication of the kernel

OpenCV Implementation

In OpenCV, the Lanczos interpolation can be performed using the cv2.resize() function. Its python implementation is as follows

image_scale = cv2.resize(
                        image_input,
                        dsize=(no_cols_output, no_rows_input),
                        interpolation= cv2.INTER_LANCZOS4                       
                           )

where dsize is the size of the output image. In this function, the kernel size used is 8x8. We rescale the input image by a scale factor of 3. The interpolated smooth output image obtained is as follows.

As we can observe the image pixel density has been increased 3x times and the numbers now appear smoother in this modified image. Now when we apply the OCR on this smoothened image, we get back the correct text value of 166.3.

print(pytesseract.image_to_string(img_scale_bbox, config=custom_config))
# This produces the following output
# 166.3

Conclusion

In many scenarios, the real world datasets that we will come across are likely to have a wide range of issues. Poor resolution and noise is one of the most important among these problems. Using the appropriate interpolation to smoothen the image to improve its quality will help improve performance of OCR engines working on these images.

Recently, I switched over to Visual Studio Code (VS Code) which is slowly becoming my favourite IDE. It somehow strikes the perfect balance between working out of the box and customizability. Further, VS Code has strong interoperability across multiple languages. I recently had to start remote development in EC2 and I was worried I might not be able to customize VS code to perform remote development. However, it turned out that VS Code provided the best framework for such remote development through ssh. I’m documenting here the steps I look to ensure a seamless access to EC2 through VS Code in Windows 10.

Machine and Software Information

Local Machine: Windows 10

Remote Machine: Ubuntu 18.04 in AWS EC2

Steps to be Followed

1. First, you need to install the Remote-SSH extension.
2. For this extension, to work we will need an OpenSSH compatible SSH Client. Unfortunately, PuTTY does not satisfy this requirement and as a result we need to install Windows OpenSSH Client.

3. We can check if the client is installed in the local machine through the following windows power shell command.

    Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'
    # which should return output that appears something like
   
    Name  : OpenSSH.Client~~~~0.0.1.0
    State : NotPresent
    Name  : OpenSSH.Server~~~~0.0.1.0
    State : NotPresent
   

4. If not present, we can install them using the following commands.

    # Install the OpenSSH Client
    Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
   
    # Install the OpenSSH Server
    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
   
    # Both of these should return output like:
   
    Path          :
    Online        : True
    RestartNeeded : False
   

5. Now we need to configure the EC2 instance to be able to connect to it. From the command palette (which can be accessed through F1 or Ctrl+Shift+P), select Remote-SSH: Open Configuration File as shown below.

   

6. You need to access the ssh config file which is typically in C:\Users\<UserName>\.ssh\config. Add the following information to the config file.

    Host ec2-aws
        HostName <hostname>
        User ubuntu
        IdentityFile `C:\Users\<UserName>\aws_keys\test-key-pair.pem
   

   The Host is a name you provide to identify the connection. The <hostname>is unique for an EC2 instance. This information can be obtained from browser based client or command line interface (CLI). Typically, I use an Ubuntu machine and connect to the machine with the username Ubuntu. An important thing to remember is that the *.pem file that is used to connect to the AWS needs to have restricted permissions. Without such restricted permission, VS Code will not be able to use it to connect to EC2. So, its best to place it within C:\Users\<UserName>\ so it can derive restricted permissions.

7. Upon configuration, we can connect to the EC2 instance through the Remote-SSH: Connect to Host as shown below

   

Conclusion

It is also a good practice to idenity the type of Remote machine (which in my case is Linux), since I think VS Code does customization based on this information. Now VS Code will provide you access to all the files/folders remote machine which you can use to do seamless remote development.