Mechanical Sympathy - Closing post 2021

 

One of my techno-geek followers sent a note a  few weeks back asking for updates on my last post for this year and asked about my favorite post. Unfortunately, I have not been actively blogging in 2021, said that I  have been working offline with the engineering community actively on exploring new technologies. One of my favorites has been learning the machine learning area.  I would like to thank all my fellow engineers for their collaboration and engagement over the years. 

As we move on to 2022, I thought of looking back for my favorite post over the years and this post beats all others and is still my favorite. 

Quote posted around four years back from Jackie Stewart "https://www.metahat.net/2018/03/mechanical-sympathy-learning-from-racing.html"

“You don’t have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy.” 

– Jackie Stewart, racing driver

I wish all of you, Happy and Healthy New Year!!!

Why do we use np.random.seed()

This is a simple function but to my surprise lot of ML engineers had asked me about the actual usage of this function and why do I use it when i am working on an algorithm which will be shared with other engineers. 

The simple answer is :

the function generates a random deterministic number. It generated the same random number each time for a specified seed  

"It's a random function with deterministic output for a specified seed. This helps to share the code and get the same output each time. "

Try this :

np.random.seed(1)

np.random.randint(low = 1, high = 10, size = 20)

The output will be, each time till seed is specified as "1" 

array([6, 9, 6, 1, 1, 2, 8, 7, 3, 5, 6, 3, 5, 3, 5, 8, 8, 2, 8, 1])

Installing Octave in Ubuntu running on Docker

I want to run Octave on Ubuntu running on docker.  Simple stuff but still manage to get errors and had to spend  few minutes on fine-tuning 

Here it goes

1- Start docket and get Ubuntu image from repo

    - Couple of commands

            - docker pull ubuntu

            - docker run -it ubuntu 

2- Install Octave 

    - ran command "apt-get install octave" and here it goes error "Unable to locate package Octave"

  

3- to come over this issue run below commands

    - apt-get update

    - apt-get install octave 

Done......

Integration Design Best Practice

Over the past few years of working on multiple critical systems solution design initiatives, there is one important thing I have learnt. 

" There is a reason we call it solution design and not system design". With the influx of cloud offerings and services, the organization no more rely on one system to automate the business process. Rather business process now relies on multiple systems performing a very specific step of the business process. All these systems are integrated to provide business process automation. This makes integration, not one of the factors but a very critical backbone of the business process." 

Nothing that I said is new and no one will disagree but unfortunately, integrations are still given less priority and design thought. 

I keep getting one question all the time " What are integration design best practices and how can I design a good reliable integration".  I have a long list of to-dos for designing good integration but sometimes being too prescriptive is not useful as it hinders a thinking ability of an architect.  

So if you are still reading and curious to know how to design a good integration than there is only one fact you need to be aware of, this will influence your design decision and push you to design better integrations. 

AND THAT IS 

Design with the above fact in mind and try to make your integration overcome failure and you will have a good design in place. Please don't spend time on developing integrations that will not fail. Such a design is purely a myth. 

Now let's go one more level down, I have put down design pillars which will help to design the integration.   These design pillars apply to any software development initiatives 

 

if this is helpful, do leave our comment. I would love to hear your design learnings. 

No supported authentication methods available (server sent: publickey)

---------------------------
PuTTY Fatal Error
---------------------------
PuTTY Fatal Error Disconnected: No supported authentication methods available (server sent: publickey)
---------------------------


Scenario : 
Error while connecting to AWS AMI Instance using PuTTy.


Solution:
in my case "admin" user worked.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html#TroubleshootingInstancesConnectingPuTTY

Docker Error :In the default daemon configuration on Windows, the docker client must be run elevated to connect

error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/auth: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

Fix :

Try one of these:

First :https://success.docker.com/article/docker-for-windows-fails-with-a-daemon-not-running-message

Second:

Dun docker service with elevated privilege. This error shows up when docker service  is not running. Make sure you docker dashboard shows as "running"

Guide for AWS Cloud Security, Governance, and Compliance

A really good collection to learn and understand basic of AWS Cloud Security, Governance, and Compliance

https://d1.awsstatic.com/training-and-certification/ramp-up-guides/RampUp_Security_102019_final.pdf

Could not switch roles using the provided information. Please check your settings and try again. If you continue to have problems, contact your administrator.

Could not switch roles using the provided information. Please check your settings and try again. If you continue to have problems, contact your administrator.

If you get this error than there is a big probability the user has no permission to assume a role. Add a new policy to user to allow assuming roles. check the below article:

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html

if it doesn't work leave a comment. 

ECS Fargate ERROR : CannotPullContainerError: Error response from daemon

Last week i was asked to look into an issue faced by the team working on a service deployed in ECS Fargate

ERROR

CannotPullContainerError: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.us-east-x.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Details:
The container service was failing to provision and stopping with the above error.


Possible Reasons:
The error indicated that the container is failing during "Pull" event and in this case, the only pull request which is configured is for the docker image maintained in ECS registry. 

Most of the times such issue occurs due to lack of access and this case was no different.. There are two possible ares to look at for enabling Fargate resource deployed in private subnet to pull image from ECS registry

• If task is launched without an public IP, its it required to configure route table on the subnet with has "0.0.0.0/0" going to a NAT Gateway or NAT instance . This is to ensure it can connect to internet. If task is launched with an public IP, configure route table on the subnet to have "0.0.0.0/0" going to an internet gateway to ensure traffic can flow in.

• Ensure the security groups for the Task allows for outbound access. 

if internet  access is a concern than another option is to deploy registry inside VPC.

Here is link for one of the option 

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/vpc-endpoints.html

OAUTH 2.0 in less than 5 min

As you had requested, here is a short video on OAUTH 2.0. More to come stay tuned.

GITLAB commands - High Frequency

I always notice the developers struggling to get the command line right for GIT. I was planning to put together list of frequently used command  but came across cheatsheet put by GitLab

https://about.gitlab.com/images/press/git-cheat-sheet.pdf

If you feel there are more which is good to have handy, just write a comment.

Happy Versioning!!!

T and Z in DateTime like 2009-08-28T23:55:16.000Z

A very simple question for a person who codes(hopefully) but for business users this can be very confusing.

What do you mean by below DateTime

2009-08-28T23:55:16.000Z

Lets explain this 

ISO 8601 (The International Standard for the representation of dates and times)
states the following definition

The "T" corresponds to Time of the date(Duhh... no brainer)  and "Z"  is special UTC designator

So the DateTime  2009-08-28T23:55:16.000Z states

Date as 2009-08-28
Time as 23:55:16.000
"Z" stands for UTC

The above time can be also be represnted in EST as
2009-08-28T23:55:16-05:00 (EST) where  -05:00 is Timezone offset


Happy coding!!!

- Metahat

Parsing Log/Text files

One of the team member was looking for ways to parse through logs to find errors in a big dump of files. I had to go to basics to get the results

Problem Statement: Find all the entries for "Error" and "failed"  in big dump of CSV/Log files. 

Solution: Old school technique ( Use Log parser)

run below commands 

C:\Program Files (x86)\Log Parser 2.2>LOGPARSER "Select Text INTO ERROR.csv from
 D:\xyz\*.log* where Text like '%ERROR%'" -i:TEXTLINE -q:Off

Statistics:
-----------
Elements processed: 14128022
Elements output:    9845
Execution time:     81.36 seconds (00:01:21.36)


C:\Program Files (x86)\Log Parser 2.2>LOGPARSER "Select Text INTO failed.csv fro
m D:\xyz\*.log* where Text like '%failed%'" -i:TEXTLINE -q:Off

Statistics:
-----------
Elements processed: 14128022
Elements output:    2025
Execution time:     76.43 seconds (00:01:16.43)

Profile not found. Tried ~/.bashrc, ~/.bash_profile, ~/.zshrc, and ~/.profile.

Error when installing NPM

 Profile not found. Tried ~/.bashrc, ~/.bash_profile, ~/.zshrc, and ~/.profile.



Fix:
Run command and try again
touch ~/.bash_profile 


Reason

https://github.com/nvm-sh/nvm/issues/1837

your system may not have a [.bash_profile file] where the command is set up. Simply create one with touch ~/.bash_profile and run the install script again

you might need to restart your terminal instance. Try opening a new tab/window in your terminal and retry."

Maven tool is not set in Jenkins pipeline

Maven tool is not set in Jenkins pipeline


Scenario
Error when building the Jenkins pipeline  (Pipeline project style)

Error snippet:
+ mvn clean package
/var/jenkins_home/workspace/atmosphere pipeline/spring-boot-samples/spring-boot-sample-atmosphere@tmp/durable-7c163949/script.sh: 1: /var/jenkins_home/workspace/atmosphere pipeline/spring-boot-samples/spring-boot-sample-atmosphere@tmp/durable-7c163949/script.sh: mvn: not found


Solution
Use the below script to refer to Maven


node 

{
def mvn_version = ''
    
withEnv( ["PATH+MAVEN=${tool mvn_version}/bin"] ) 
  {   
      sh  'mvn clean package'
  }

}


Image of Jenkins Home ->Manage Jenkins ->Global Tool Configuration

FOR FREESTYLE PROJECT ( REFER TO MAVEN ERROR)

SSH into a running Linux container - docker

Looking for extensive set of commands, refer
http://phase2.github.io/devtools/common-tasks/ssh-into-a-container/


 In summary, use below commands as specified in article

• Use docker ps to get the name of the existing container
• Use the command docker exec -it {containername} /bin/bash to get a bash shell in the container
• Generically, use docker exec -it {containername} {command} to execute whatever command you specify in the container.

running docker as root user 

docker exec -it --user root {container id} /bin/bash

Cannot run program "mvn" (in directory : error=2, No such file or directory

Note : In below scenario jenkins was deployed in Linux container

Scenario:  Jenkins build failed with error.

java.io.IOException: error=2, No such file or directory

 at java.lang.UNIXProcess.forkAndExec(Native Method)
 at java.lang.UNIXProcess.(UNIXProcess.java:247)
 at java.lang.ProcessImpl.start(ProcessImpl.java:134)
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
Caused: java.io.IOException: Cannot run program "mvn" (in directory "/var/jenkins_home/workspace/atmosphere"): error=2, No such file or directory



Fix:
The issue was due to absence of Maven files in container. To deploy Maven files , follow these steps:


Go to Global Tool Configuration (http://localhost:xxxx/configureTools/)
or navigate Jenkins Home Page - > Manage Jenkins - > Global Tool Configuration
Scroll down and make the below configuration under Maven Section
Click Save





 


















In Jenkins , go to teh Jenkins job and select configure

















scroll down to BUILD Section and select MAVEN_HOME from dropdown





















Rebuild Project

Debugging the Azure Event Grid based Azure Functions locally using Visual Studio

If you want to debug the Azure function locally in visual studio which are triggered by resources like Azure Blog storage e.g. when a file is created in Azure Blog storage account than you are in right place

Check this out

Azure Function Event Grid Trigger Local Debugging

Bad IL format - Azure function

First, try restarting the Azure Function app. 

The error generally shows up when changes are made to the function

Azure Durable Functions - Good Starter Video

Azure Durable Functions for Serverless .NET Orchestration