Digital Solutions https://www.neurealm.com/category/blogs/digital-solutions/ Engineer. Modernize. Operate. With AI-First Approach Mon, 29 Sep 2025 13:08:51 +0000 en-US hourly 1 https://wordpress.org/?v=6.8.3 https://www.neurealm.com/wp-content/uploads/2025/05/Favicon.svg Digital Solutions https://www.neurealm.com/category/blogs/digital-solutions/ 32 32 The Importance of ESG for Tech Companies https://www.neurealm.com/blogs/the-importance-of-esg-for-tech-companies/ Sat, 26 Oct 2024 04:43:15 +0000 https://gavstech.com/?p=13619 The post The Importance of ESG for Tech Companies appeared first on Neurealm.

]]>

Embracing Environmental, Social, and Governance (ESG) principles involves adopting sustainable practices, addressing social impact, and ensuring effective governance. ESG-driven innovation in energy efficiency, sustainable materials, and responsible AI development positions tech firms for new competitive advantages and market opportunities. In today’s world, ESG factors are becoming increasingly important for businesses, particularly in the tech industry.  Companies now must consider these broader sustainability factors for long-term success.

Benefits of ESG for Tech Companies

Enhanced Reputation and Brand Image: Implementing ESG practices in tech companies can enhance their reputation and brand image. Demonstrating a commitment to sustainable and socially responsible practices contributes to a positive perception among customers, investors, and the broader public. This positive image can differentiate the company in a competitive market, attracting socially conscious consumers and fostering brand loyalty.

Stronger Relationships with Stakeholders: ESG initiatives strengthen relationships with various stakeholders, including employees, customers, suppliers, and communities. Tech companies prioritizing environmental sustainability, social responsibility, and ethical governance build trust and credibility. Engaging with stakeholders on ESG issues allows for meaningful collaboration, feedback, and shared values, creating a sense of purpose and alignment among all involved parties.

Improved Access to Capital: Adopting robust ESG practices can enhance a tech company’s access to capital. Investors, including institutional and Socially Responsible Investment (SRI) funds, increasingly consider ESG factors in their decision-making processes. Companies with strong ESG performance may attract a broader investor base, potentially lowering the cost of capital. Additionally, there is a growing trend of sustainable finance and green bonds, providing capital for environmentally friendly projects.

Reduced Operational Costs: Efficient ESG practices often reduce operational costs. Implementing energy-efficient technologies, waste reduction measures, and sustainable supply chain practices contribute to cost savings. Tech companies can benefit from lower energy bills, optimized resource utilization, and streamlined processes, resulting in financial savings and a smaller environmental footprint.

Attracting and Retaining Talent: Tech companies with a strong ESG focus can attract and retain top talent. Many employees, especially younger generations, seek employers committed to environmental and social responsibility. A positive ESG reputation contributes to a positive workplace culture, promoting employee satisfaction, retention, and recruitment, which is vital in the competitive tech industry.

Steps To Implement ESG Principles

Setting Clear ESG Goals

Conduct a Materiality Assessment: Conduct a materiality assessment to identify the most relevant environmental, social, and governance factors for your tech company. Consider inputs from internal and external stakeholders to prioritize key issues.

Define Specific and Measurable Goals: Set clear and measurable ESG goals aligned with your company’s values and priorities. Ensure these goals are specific, time-bound, and quantifiable to track progress effectively.

Align with Global Standards: To enhance transparency and comparability, consider aligning ESG goals with widely recognized frameworks such as the Global Reporting Initiative (GRI), Sustainability Accounting Standards Board (SASB), or Task Force on Climate-related Financial Disclosures (TCFD).

Integrating ESG into Decision-Making

Embed ESG in Corporate Strategy: Integrate ESG considerations into the overall corporate strategy. Ensure that ESG goals are not treated as separate initiatives but are woven into the fabric of decision-making processes.

Include ESG Criteria in Product Development: Consider ESG factors in developing products and services. This could involve assessing the environmental impact, ensuring ethical supply chains, and designing technology with social considerations.

Incorporate ESG into Risk Management: Identify and assess ESG-related risks and opportunities. Integrate this information into the company’s risk management processes to mitigate risks and capitalize on opportunities.

Reporting on ESG Performance

Implement Robust Data Collection: Establish a systematic process for collecting relevant ESG data. This may involve working with various departments to gather information on energy consumption, diversity and inclusion metrics, ethical sourcing, and other key indicators.

Choose Appropriate Reporting Standards: Select recognized ESG reporting standards that align with your industry and business model. Consistent reporting allows for easier comparison and benchmarking.

Regularly Communicate Progress: Regularly update ESG performance through sustainability reports, annual reports, or integrated reporting. Communicate achievements, challenges, and future targets to stakeholders.

Engage Stakeholders in Reporting: Engage with stakeholders to gather their perspectives on ESG performance and reporting. This collaboration fosters transparency and ensures reporting reflects a holistic view of the company’s impact.

Leverage Technology Solutions: Explore technology solutions and software platforms that facilitate ESG data management, analysis, and reporting. These tools can streamline the process and improve accuracy.

Future of ESG for Tech Companies

The future of ESG for tech companies entails a deeper integration of sustainability, social responsibility, and governance into core business strategies. Tech firms will focus on innovative solutions for environmental challenges, prioritize ethical AI, and enhance diversity and inclusion. Regulatory pressures will increase, necessitating robust governance structures. ESG metrics will be standard in financial reporting, shaping investor decisions. Collaboration with stakeholders will be paramount for holistic ESG success. As society demands more responsible technology, companies proactively addressing ESG considerations will not only future-proof their operations but also gain a competitive edge, fostering a positive impact on the industry and global sustainability efforts.

Neurealm is committed to ESG responsibilities and works across several areas to contribute to the social and environmental needs of the communities we live in. Our commitment to ethical conduct goes hand in hand with our business goals, and we invest quite heavily in local communities across geos. Our strategy includes social and environmental goals along with our financial goals. We strongly believe that social and environmental sustainability programs should go hand in hand with our mainstream business. For more information, please visit https://Neurealmtech.com/about-us/corporate-social-responsibility/.

The post The Importance of ESG for Tech Companies appeared first on Neurealm.

]]>
The Disruption Caused by Generative AI https://www.neurealm.com/blogs/the-disruption-caused-by-generative-ai/ Mon, 21 Oct 2024 04:39:23 +0000 https://gavstech.com/?p=13614 The post The Disruption Caused by Generative AI appeared first on Neurealm.

]]>

Generative Artificial Intelligence (GenAI) is a disruptive force reshaping industries. The potential disruption caused by GenAI is multifaceted and can be felt across various sectors and aspects of society. GenAI’s profound impact on society necessitates robust regulation, ethical frameworks, and proactive adaptation to harness its benefits while mitigating potential disruptions.

Job Market Transformation

Automation of Tasks: GenAI heralds a significant transformation in the job market, primarily through task automation. GenAI has the potential to automate numerous knowledge-based tasks currently undertaken by humans. Sectors like content creation, translation, data entry, financial analysis, and software development may witness job displacement as GenAI systems become proficient in handling these responsibilities. This shift raises questions about the redefinition of roles and the need for upskilling to align with the evolving demands of the automated job landscape.

New Job Creation: While the automation of specific tasks may result in job displacement, it is anticipated that GenAI will also stimulate the creation of unique and specialized roles. Emergent professions may include AI trainers responsible for fine-tuning and enhancing AI capabilities, ethicists focusing on the responsible development and use of GenAI, data scientists specializing in GenAI applications, and professionals engaged in managing and overseeing collaborations between humans and AI systems. The dynamic evolution of the job market, spurred by GenAI, necessitates a proactive approach to education, training, and adapting skill sets to thrive in this transformative era.

Content Creation and Media Landscape

Deepfakes and Misinformation: GenAI’s proficiency in generating realistic audio, video, and text content raises significant concerns regarding the proliferation of deepfakes and the spread of misinformation. The ability to fabricate convincing content can erode trust in media, create societal challenges, and complicate the verification of information. This necessitates the development of robust tools and strategies to detect and counteract the impact of deepfakes, ensuring the integrity of information sources and safeguarding against malicious uses of this technology.

Personalized Content and Marketing: GenAI’s capabilities enable the creation of personalized content tailored to individual users. In fields such as education, entertainment, and advertising, this leads to more engaging and effective marketing strategies. By customizing content based on user preferences, behaviors, and demographics, GenAI enhances user experiences, potentially increasing audience engagement and the effectiveness of targeted marketing efforts. However, ethical considerations, including user privacy and consent, must be prioritized to ensure responsible and transparent implementation of personalized content strategies.

Scientific Discovery and Innovation

Accelerated Drug Discovery: Discovering and developing new drugs traditionally involves time-consuming and expensive steps, including target identification, compound screening, and clinical trials. This often takes years, and many potential drug candidates may fail at various stages. GenAI can significantly accelerate drug discovery by utilizing machine learning models to generate and evaluate potential drug molecules. By predicting the properties, efficacy, and possible side effects of various compounds, GenAI can streamline the selection process, reducing the time and resources required for preclinical and clinical testing. GenAI can simulate and analyze vast numbers of molecular structures, allowing for high-throughput screening in silico. This enables researchers to identify promising drug candidates more efficiently, potentially discovering novel treatments for various diseases.

Novel Material Design: Developing new materials with specific properties involves a complex interplay of various factors. Traditional experimental methods can be time-consuming and costly, limiting the exploration of a wide range of possibilities. GenAI can contribute to material design by generating and analyzing virtual structures, predicting their properties, and proposing novel combinations of elements. This accelerates the discovery of materials with tailored characteristics, such as improved conductivity, strength, or flexibility. The ability to design materials with enhanced properties has applications in renewable energy technologies, battery development, and aerospace. For example, lightweight and durable materials prepared with the help of GenAI can contribute to creating more efficient solar cells, longer-lasting batteries, and stronger yet lighter components for aircraft.

GenAI capabilities may surpass its initial design as it continues to learn and evolve. This raises questions about who controls its development and the potential risks associated with unintended consequences. For example, Getty Images filed a lawsuit against Stability AI, the company behind the Stable Diffusion image generation model, alleging the model infringed on Getty Images’ copyrights by using millions of images from their website without permission to use them to train the model. This case highlights the potential for ownership and control issues in GenAI regarding the training data used to develop the models. The outputs generated by GenAI, whether creative work or solutions, raise questions about intellectual property rights. Determining ownership becomes essential for fair compensation, acknowledgment, and legal responsibilities. Ensuring responsible development and use of GenAI involves defining ethical application guidelines, preventing misuse, and promoting the common good. Establishing legal frameworks that govern ownership, control, and responsible use of GenAI-generated content is imperative. This includes addressing issues of liability, accountability, and potential societal impacts.

With a dedicated GenAI expert team, deep understanding of analytical AI, a strong data science and cybersecurity background, Neurealm brings a structured approach to address diverse challenges using GenAI. We partner with you to build solutions specific to your use cases to bring tangible benefits to your customers and employees, responsibly and securely. To create a ready pool of GenAI certified professionals, ZIFGENAI University – a part of Neurealm’ Learning Academy has initiated a Generative AI skilling project to empower employees to learn new, job-relevant skills.

You can find more information on our Generative AI services at https://Neurealmtech.com/services/digital-solutions/generative-ai/.

The post The Disruption Caused by Generative AI appeared first on Neurealm.

]]>
Unlocking Project Management Secrets Using AI https://www.neurealm.com/blogs/unlocking-project-management-secrets-using-ai/ Fri, 24 May 2024 06:10:29 +0000 https://gavstech.com/?p=14515 The post Unlocking Project Management Secrets Using AI appeared first on Neurealm.

]]>

Project management isn’t confined to software engineering; it originated in non-software industries like manufacturing and evolved to encompass software development. The process can involve building new products, enhancing existing software, or integrating different solutions. Over time, various models emerged, including the traditional waterfall and agile methodologies, with hybrid approaches adapting to project requirements and complexity. Projects can be categorized based on their focus, whether on research, cost, or time constraints.

In recent years, AI has entered the realm of project management. AI in project management is useful in handling data-driven decision-making and tasks such as progress tracking, cost management, and scheduling. Enhanced with AI capabilities, platforms like Microsoft Project can streamline tasks like resource allocation and dependency management.

Neurealm conducted a webinar, Everything Products Episode 9: Predicting the Unpredictable : Unlocking Project Management Secrets Using AI, with Ms. Anaya Bakshi Associate Director Engineering, Neurealm and Rahul Bidwe Associate Manager Engineering, Neurealm as the panelists.

Understanding Project Management

Project management involves strategically coordinating skills and techniques to accomplish tasks, planning, organizing, and overseeing projects to meet specific goals. Project managers, seen as stewards of projects, must create a collaborative team environment, engage stakeholders, focus on project values, and navigate complexities while optimizing risk responses. Adaptability and embracing change are critical, along with adhering to PMI’s 12 principles and four values: responsibility, respect, fairness, and honesty. This role goes beyond technical expertise, requiring strong leadership and multifaceted skills.

Traditional ML vs. Emerging Generative AI

Traditional machine learning typically involves analyzing historical data to identify patterns and make predictions based on statistical analysis. In contrast, generative AI, particularly evident in recent advancements like OpenAI’s GPT, focuses on generating new content based on the input data it receives. Generative AI’s strength lies in Natural Language Processing (NLP), making it adept at understanding and producing human-like text.

While traditional ML has been effective in predictive analytics, generative AI expands its capabilities into face recognition, emotional recognition, and sentiment analysis. AI, particularly generative AI, enhances and generates new content based on input data, offering potential for innovative applications across various domains.

Importance of AI in Project Management

AI can quickly analyze numerical and statistical data, identify anomalies, and provide insights for informed decision-making. Additionally, AI can assist in budgeting, financial planning, and identifying key cost drivers, especially in rapidly changing technology domains. However, there is also a need for human intervention in risk management and project planning, as AI is still evolving in these domains. AI presents promising opportunities to enhance project management efficiency and effectiveness, but human expertise remains crucial in certain critical aspects of project management.

Uses of GenAI in Project Management

Generative AI can assist project managers in areas such as the following:

Automation: AI can handle simple tasks like summarizing meeting notes or generating standard reports with minimal or no human intervention. This saves time and ensures accuracy in routine tasks.

Complex activities with human intervention: In tasks such as cost-benefit analysis, scheduling plans, or risk analysis, AI can provide initial drafts based on prompts. Project managers can then review these drafts, incorporate expert opinions, and make necessary adjustments according to project requirements.

Augmentation for complex tasks: In more intricate activities like building a business case with multiple interdependencies and variables, AI can assist in providing initial data and supporting with interdependencies. While the bulk of the work still requires human decision-making and expertise, AI can streamline the process by handling repetitive tasks and providing valuable insights.

Leveraging AI in Product Engineering

AI, particularly generative AI, can streamline complex tasks in product engineering, particularly in tools like Jira. AI simplifies and accelerates various aspects of product engineering, significantly reducing the time and effort required for project management activities. Here are some use cases:

Seat allocation in hybrid working models: With limited seats and rotating teams, AI can analyze statistical data and generate multiple scenarios to allocate seats efficiently, reducing conflicts and simplifying decision-making.

Portal development in Jira: AI can generate epics and user stories in Jira based on input data, streamlining the portal development process. It creates user stories for database population, UI design, and backend development, reducing the time and effort required for brainstorming and planning.

Release planning: AI identifies interlinked stories and creates release plans, determining when minimum viable products can be released to consumers, thereby optimizing release management workflows.

Test plan creation: AI reviews each story and generates test plans, highlighting stories with lower testing coverage. Thus, comprehensive testing coverage is ensured without the need for manual intervention.

Reporting: AI analyzes data to create velocity reports, release-based burndown charts, and engineering-based performance reports, providing insights into project performance and reducing the time spent on manual reporting tasks.

This blog offers only a high-level gist of the webinar. However, you can watch the entire discussion here. For more such videos, pl. visit https://www.gslab.com/webinars/ and https://www.Neurealmtech.com/videos/.

Neurealm engineers digital products that scale and support business transformations at record speed to compete in the digital world. Our engineering teams craft new products and functionalities that ensure speed to value while keeping operational costs low. For more on our digital product engineering services, please visit https://Neurealmtech.com/services/digital-engineering/digital-product-engineering/.

The post Unlocking Project Management Secrets Using AI appeared first on Neurealm.

]]>
Machine Learning Makes Migration from Bangalore to Pune Simpler https://www.neurealm.com/blogs/machine-learning-makes-migration-from-bangalore-to-pune-simpler/ Wed, 08 May 2024 05:23:07 +0000 https://gavstech.com/?p=14225 The post Machine Learning Makes Migration from Bangalore to Pune Simpler appeared first on Neurealm.

]]>
Inspiration

A friend of mine is a native from Bangalore, Karnataka. Recently, for some personal reasons, she decided to move to Pune. She called me to check what neighborhoods in Pune are like her current neighborhood in Bangalore. She loves her current neighborhood in Bangalore. She went on and on about different amenities, facilities, and life in her neighborhood. She would like it to close to her office when it starts. While I know Pune reasonably well, I quickly realized that no one person would have the details. Everyone has biases and preferences. Wouldn’t it be great if she can find the neighborhoods in Pune that are exactly like his current neighborhood in Bangalore? She can then pick a neighborhood like her current neighborhood and very close to her new workplace.

I knew the data for such information exists. The data scientist in me got excited. Some of the machine learning enthusiasts from Neurealm and I discussed this problem. Here, we came up with a neighborhood recommender. With this, now anyone moving between the cities can choose a neighborhood of liking.

How does it work?

It is simple. Pick cities, neighborhoods, decide the number of venues to be compared, and let Machine Learning kick in to recommend matching communities. It shows you the map. It lists all the points of comparison. Isn’t it cool?

If you are curious to know what is under the hood of this recommender, read on.

Data Description
Methodology

We use Foursquare APIs to get a list of venues nearby these neighborhoods. We have scraped data for categories like Food, Parks, Schools, Libraries, Theatres, Nightlife, Shops, and Grocery stores, etc. Here is a sample table of such venue information:

We have done some exploratory data analysis by analyzing data sets to summarize their main characteristics with visual methods. We have done this on combined data (Pune and Bangalore) for comparison purposes.

[vc_single_image image=”6896″][vc_empty_space height=”25px”]
Statistical Analysis Technique

We used the K-means clustering technique to analyze the data. K-means algorithm is an iterative algorithm that tries to partition the dataset into K pre-defined distinct non-overlapping subgroups (clusters) where each data point belongs to only one group. It tries to make the intra-cluster data points as similar as possible while also keeping the clusters as different (far) as possible.

Results

We applied the K-means clustering technique on data separately (on Pune and Bangalore neighborhood data) in python.

Elbow Method:

K-means is somewhat naive — it clusters the data into k clusters, even if k is not the correct number of clusters to use. When we come to clustering, it is hard to know how many clusters are optimal.

One method to validate the number of clusters is the elbow method. The idea of the elbow method is to run k-means clustering on the dataset for a range of values of k (say, k from 1 to 10), and for each value of k, calculate the Sum of Squared Errors (SSE).

When we graph the plot, we see that the graph levels off slowly after three clusters. This implies that the addition of more clusters will not help us that much.

Scatter Graph:

Let us visualize the results by plotting the data colored by these labels. We will also plot the cluster centers as determined by the k-means estimator.

In the scatter plot, we have used three categories for both the cities as suggested by the elbow curve.

Cluster Analysis:

We have created three clusters as

Bangalore neighborhood clustering indicates that there are three clusters, namely

  • Cluster – 0: The neighborhoods fall under this category: Food, Parks, Grocery Stores, and Theatres.
  • Cluster – 1: The neighborhoods fall under this category: Grocery Stores, Food, Parks, and Libraries.
  • Cluster – 2: The neighborhoods fall under this category: Schools, Foods, and Grocery Stores.

Pune neighborhood clustering indicates that there are three clusters, namely

  • Cluster – 0: The neighborhoods fall under this category: Food, Theatres, and Schools.
  • Cluster – 1: The neighborhoods fall under this category: Food, Grocery Stores, and Libraries.
  • Cluster – 2: The neighborhoods fall under this category: Grocery Stores, Foods, and Libraries.
Mapping Clusters:

We can map these neighborhood clusters as:

Based on the above plots we can match clusters for Pune and Bangalore as

 

We can conclude that the person living in Cluster – 0 of Bangalore should live in Cluster – 2 of Pune. For example, the person living in Jayanagar, Bangalore, should live in Aundh, Pune because there is nearly the same number of Grocery Stores, Foods, and Parks in both the clusters. Also a significant number of Parks and schools. So based on the analysis, we can suggest the person move from Jayanagar, Bangalore to Aundh, Pune.

Based on the top 10 venues, we recommend Aundh Pune, which has top 10 venues of:

Aundh, Pune 18.5602 73.80310 Grocery Stores School Library Department Store Parks Coffee Shop Chinese Restaurant Bar Shopping Mall Gas Station

Which we can easily see to be comparable with top venues of Jayanagar Bangalore top venues of:

Jayanagar, Bangalore 12.9308 77.5838 Grocery Store Library Park Indian Restaurant Multiplex

With venues like Grocery Stores, Libraries, Parks, Restaurants in common, the transition for my friend is going to be a very smooth one!

Discussions

Bangalore has 76 unique venue categories, with different neighborhoods having a different number of venues and some neighborhoods even reaching 50 for the number of venues. There are quite a few Coffee Shops, a variety of restaurants, parks, stores, etc. Pune has 76 unique venue categories. Most of the neighborhoods have reached the limit of 50, with the remaining having overall a very high count. There are various Libraries, Parks, Grocery Stores, Cineplex Schools, theatres, hotels, and plazas.

Limitations and Further Work

In this project, we have considered only the frequency of venues of a specific category to categorize neighborhoods. Many other factors can augment it (e.g., other statistical measures like variance, various percentiles, etc. of this number, cost of living, family factors, availability of housing, public transportation, etc.). We are also relying on data from foursquare for our analysis. We are also using the free developer’s account, imposing some restrictions on the data we obtain. We can enhance the study further w.r.t. both these factors as well.

Conclusion

Bangalore and Pune City are similar in their neighborhoods, being large metropolitan cities in India. A person hailing from Jayanagar is advised to pick Aundh for settling down while moving from Bangalore to Pune City. You can carry out a similar analysis for any neighborhood in these cities or extend it for migration between any other cities/countries by gathering similar data for those two cities.

Credits: Sagar Nangare and Vijay Singh

   Author

Sameer Mahajan | Principal Architect

Sameer Mahajan has 25 years of experience in the software industry. He has worked for companies like Microsoft and Symantec across areas like machine learning, storage, cloud, big data, networking and analytics in the United States & India.

Sameer holds 9 US patents and is an alumnus of IIT Bombay and Georgia Tech. He not only conducts hands-on workshops and seminars but also participates in panel discussions in upcoming technologies like machine learning and big data. Sameer is one of the mentors for the Machine Learning Foundations course at Coursera.

The post Machine Learning Makes Migration from Bangalore to Pune Simpler appeared first on Neurealm.

]]>
Using “Kudu” to Manage Azure Web Apps https://www.neurealm.com/blogs/using-kudu-to-manage-azure-web-apps/ Wed, 08 May 2024 05:22:59 +0000 https://gavstech.com/?p=14200 The post Using “Kudu” to Manage Azure Web Apps appeared first on Neurealm.

]]>
Introduction

We discussed about creating a web app on Azure using Visual Studio here and through eclipse here. In this blog, we’ll go through Kudu – a managing & monitoring service available for every App Service on Azure.

Web Apps Dashboard on Azure

Once Web App is created successfully, we can see Web App management dashboard as below.

It will list down App URL, App Service Plan & FTP Server Details. Azure will deploy the binaries to the server and provide very limited access for updating the binaries and log files through FTP.

Web Apps Management & Troubleshooting through “Kudu”

Every Azure Web App includes a “hidden” or “background “service site called Kudu.  It is useful for capturing memory dumps, looking at deployment logs, viewing configuration parameters and much more.

We can access the Kudu service through the portal by navigating to Web App dashboard > Advanced Tools > Click on Go.

Another, more direct method is to modify your web app’s URL. Specifically, you should insert the site control manager (“scm”) token into the URL, as shown below:

https://mywebappname.scm.azurewebsites.net

If you’ve mapped your own public DNS name to your web app, then you’ll still need to use the original *.azurewebsites.net DNS name to access Kudu.

Kudu console General Navigation

Following screenshot shows Azure web app’s Kudu dashboard with the Debug console and Tools menus simultaneously exposed.

The Environment page gives Azure web site administrators several pieces of valuable information:

  • App settings
  • Connection strings
  • Environment variables
  • Runtime versions
  • Server variables
  • HTTP headers

This data is enormously helpful to have, especially when you recall that Azure web apps use a platform-as-a-service (PaaS) model in which we have limited direct control of the underlying Hyper-V virtual machine(s).

Of course, that data (especially raw connection strings) is sensitive, so accessing the kudu console requires authenticating yourself as an Azure administrator.

Let’s review some other useful Kudu-based web app administrative tasks.

Retrieve Diagnostic Dump

Azure PaaS web apps run on Windows Server VMs and Internet Information Services (IIS). As you know, IIS offers verbose logging options. In Kudu, fetch the diagnostic logs by clicking Tools > Diagnostic Dump. This action yields a .zip file that contains the log data, current to their generation time.

View running processes

Click Process Explorer on the Kudu top navigation bar to see a stripped-down, web-based version of Windows Task Manager. This is a read-only view of your PaaS VM’s running processes. It is mainly useful to see if any, and which, processes consume too many resources in your web app.

Launch a Debug console

Kudu provides powerful command line access for monitoring logs and various folders under our VM. Because we don’t have full-stack access to the underlying VM the same way we do with the Azure infrastructure-as-a-service (IaaS) scenario, we can use diagnostic console to view and modify data with PowerShell & CMD.

Site extensions

Site extensions are the utilities provided by Azure for Web Apps. Those can be viewed and managed through Kudu’s Site Extension tab. Best example of such utility is “Application Insights”. It provides complete monitoring of the deployed Web App. Additional Extensions could be browsed through Gallery.

REST API endpoints

The Kudu dashboard has a REST API section that lists various service data endpoints for your web app. For instance, I can use below REST API endpoint to update war file for my Java Web Application.

Connect Kudu to specific instance of the App

Kudu site always connects to a single instance even though the Web App is deployed on multiple instances. E.g. If the site is hosted in an App Service plan which is scaled out to 3 instances, then at any time the KUDU will always connects to one instance only. However, there’s way to access Kudu for specific instance using ARRAffinity cookie.

The ARRAffinity cookie is unique and bounds to an instance. If there are multiple instances, then there will be ARRAffinity cookie corresponding to those many instances. So we can modify the ARRAffinity cookie in the HTTP response and force KUDU to connect to a specific instance, provided the cookie is enabled in Web App settings.

Conclusion

We have explored various capabilities of Kudu Service for managing and monitoring Web App in Azure App Service. Keep following our blog series where we will further explore various features of Azure Web Apps like Authentication, Deployment Slots, and A/B Testing etc.

   Author

Kedar Joshi

The post Using “Kudu” to Manage Azure Web Apps appeared first on Neurealm.

]]>
Booting CentOS from UEFI https://www.neurealm.com/blogs/booting-centos-from-uefi/ Wed, 08 May 2024 05:22:57 +0000 https://gavstech.com/?p=14208 The post Booting CentOS from UEFI appeared first on Neurealm.

]]>

Basic Input Output System (BIOS) is firmware that sits between hardware and software. BIOS initializes all hardware components like CPU, RAM, graphics cards, peripherals etc. it then performs Power On Self Test (POST) to ensure everything is working correctly. It then looks for bootable device and hands over control to the OS. Thus, BIOS has been playing a crucial role in booting a device for a long time.

However, BIOS technology is outdated. Its interface is text based blue screen as shown here:

It can boot hard drives that are 2.1 TB or less which is because it uses Master Boot Record (MBR). MBR stores the partitioning information on a drive. The organization of the partition table in the MBR limits the maximum addressable storage space of a partitioned disk. MBR also only supports up to four primary partitions.

Motivation

To overcome these limitations Unified Extensible Firmware Interface (UEFI) came up.

Advantages of UEFI
  • It can work with drives of size millions of petabytes as it uses GUID Partition Table (GPT) rather than MBR. The practical limits are dependent on the operating system and its file systems
  • It allows large number of partitions typically dependent on your operating system
  • It stores partitioning and boot data in multiple copies across the disk making it more robust and recoverable in case of data corruption
  • GPT also stores cyclic redundancy check (CRC) values to check that its data is intact. If the data is corrupted, GPT can notice the problem and attempt to recover the damaged data from another location on the disk
  • It can support graphical interface unlike BIOS text based blue screen
  • Supports secure boot and can check malware during boot
  • Supports networking features within the firmware and remote troubleshooting and configuration
Migration Steps

Following are the steps required to boot CentOS from UEFI.

Preparing EFI Bootloader

The EFI system partition needs to be formatted with a file system whose specification is based on the FAT file system and maintained as part of the UEFI specification. The EFI bootloader contains .efi file.

  • efiboot.img has file structure /EFI/BOOT/
  • Steps to create custom efiboot.img
    • #dd if=/dev/zero bs=1024 count=1440 of=efiboot-new.img        <– create a .img
    • #mkfs.vfat -n “BOOT_EFI” efiboot-new.img                                        <– format it with FAT file system
    • #mkdir new
    • #mount -t vfat -o loop efiboot-new.img new
    • #mkdir new/EFI
    • #mkdir new/EFI/BOOT/
  • Copy the configuration (like grub configuration) and efi file
  • #cp BOOTX64.efi new/EFI/BOOT/                                                            <– Prerequisite we need .efi file
  • #umount new
  • #mv efiboot-new.img efiboot.img
Working with bootloaders

EFI with GRUB

For centos 6 distribution we can get the BOOTX64.efi file

https://mirror.centos.org/centos/6/os/x86_64/EFI/BOOT/

Command to generate bootx64.efi

grub-mkimage -o bootx64.efi -p /efi/boot -O x86_64-efi

fat iso9660 part_gpt part_msdos

normal boot linux configfile loopback chain keylayouts

efifwsetup efi_gop efi_uga jpeg png

ls search search_label search_fs_uuid search_fs_file

gfxterm gfxterm_background gfxterm_menu test all_video loadenv

exfat ext2 ntfs udf password password_pbkdf2 pbkdf2 linuxefi

EFI with syslinux bootloader

The last update on syslinux community was on 2014-10-06 : Syslinux 6.03 released. The syslinux bootloader have efi support.

https://wiki.syslinux.org/wiki/index.php?title=The_Syslinux_Project

https://wiki.syslinux.org/wiki/index.php?title=Install#UEFI

Configuration for syslinux EFI bootloader

https://docs.slackware.com/howtos:slackware_admin:set_up_syslinux_as_boot_loader_on_uefi_based_hardware

Add efiboot.img bootloader to iso

The command ‘genisoimage’ have option -e to add the efibased bootloader

Example:

  • genisoimage  -V “CentOS 7 x86_64”  -A “CentOS 7 x86_64”  -o mytest.iso -joliet-long -b isolinux/isolinux.bin  -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e isolinux/efiboot.img  -R -J -v -T live
  • The ‘mkisofs’ command
  • mkisofs -U -r -v -T -J -joliet-long -V “RHEL-7.1 Server.x86_64” -volset “RHEL-7.1 Server.x86_64” -A “RHEL-7.1 Server.x86_64” -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -o ../hpbs.iso .

Make BIOS and EFI hybrid boot iso

Add efiboot as second boot image in mkisofs or genisoimage -eltorito-alt-boot -e images/efiboot.img -no-emul-boot

https://fedoraproject.org/wiki/User:Pjones/BootableCDsForBIOSAndUEFI

Example:

  • genisoimage  -V “CentOS 7 x86_64”  -A “CentOS 7 x86_64”  -o mytest.iso -joliet-long -b isolinux/isolinux.bin  -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e isolinux/efiboot.img  -R -J -v -T live

Add utility for GPT partition

gdisk – interactive utility

sgdisk – scriptable version of gdisk

Example

  • sgdisk –clear
  • –new 1::+1M –typecode=1:ef02 –change-name=1:’BIOS boot partition’
  • –new 2::+100M –typecode=2:ef00 –change-name=2:’EFI System’
  • –new 3::-0 –typecode=3:8300 –change-name=3:’Linux root filesystem’

   Author

Sameer Mahajan | Principal Architect

Sameer Mahajan has 25 years of experience in the software industry. He has worked for companies like Microsoft and Symantec across areas like machine learning, storage, cloud, big data, networking and analytics in the United States & India.

Sameer holds 9 US patents and is an alumnus of IIT Bombay and Georgia Tech. He not only conducts hands-on workshops and seminars but also participates in panel discussions in upcoming technologies like machine learning and big data. Sameer is one of the mentors for the Machine Learning Foundations course at Coursera.

The post Booting CentOS from UEFI appeared first on Neurealm.

]]>
Salesforce Configuration or Customization https://www.neurealm.com/blogs/salesforce-configuration-or-customization/ Wed, 08 May 2024 05:22:53 +0000 https://gavstech.com/?p=14212 The post Salesforce Configuration or Customization appeared first on Neurealm.

]]>
Optimize Salesforce for the best value creation

A CRM is a key functional tool of any business, linking all the stakeholders in an organization — the employees, leaders, customers, prospects, and the organization itself and Salesforce is one of the most trusted CRMs globally, with an ecosystem of 10+ products, 150,000+ customers, and 1,800+ partners.

So what makes Salesforce such a popular CRM platform?

Primarily, Salesforce is widely adopted as it is highly configurable, customizable, robust, and flexible. However, every organization has its different operations and its own unique need and approach to use Salesforce. Therefore, Salesforce shouldn’t be treated as a plug-and-play model, where you get it up and running as soon as it is bought. Rather, an organization should consider the services of a Salesforce expert who will configure, customize, and implement it according to the organization’s unique needs.

The organizations who decide to have Salesforce as their CRM are often in a dilemma about choosing the right method. To optimize the various features available with Salesforce, an organization has to choose either a Salesforce configuration or a Salesforce customization, or at times a combination of both methods. When implementing Salesforce, almost 90% of businesses deal with both methods.

But which one is best suited for your organizational needs?

Read on to understand how these methods work, and when and how to choose either of the methods or a mix of both.

Understanding the nuances of Salesforce Configuration

The Salesforce configuration method is quick, fast, and accessible as it works within the system itself. Configuration mostly means doing it yourself, obviously with some expert help that will make it a timely implementation.

An organization does not need to customize the solution but make it run using what is provided. In Salesforce language, we call it “using the user-defined values to create a certain module or feature.”

To emphasize, an organization will buy a pre-defined application by Salesforce, set up initial settings, integrate new fields, modules, surveys, workflows, and reports by simply altering settings in the Salesforce panel with help from experts, and get going quickly.

However, there are some features that you can configure in Salesforce such as adding objects/fields, adding tabs/apps, creating profiles/setting permissions, workflows and custom reports, validation rules, and email templates.

Getting to know Salesforce Customization better

Salesforce customization means enhancing or altering the original Salesforce application to align with the exact needs of the organization. This process is one step ahead of the configuration, which allows you to bring in features and functionalities that are not part of the application.

Now, Salesforce customization is tedious and time-consuming, but it will deliver your expected outcome of Salesforce seamlessly. Also, it is important to adhere to all Salesforce best practices and guidelines while customizing it. This ensures the customizations are migrated without issues to Salesforce platform releases that usually occur ~3 times a year.

Consider this example.

An organization will buy Salesforce and custom code it to have features that may not be possible to achieve through configuration. The Salesforce customization will require Salesforce expertise and is a complex process, mostly followed by organizations with a large user base.

With a customization, you can build and customize Apex classes/triggers, Visual Force pages/components/emails, lead capture forms, third-party integration, lightning components, and partner portals.

Salesforce configuration or Salesforce customization
Configuration or Customization — the Dilemma of Selection

Before you finally zero in on the best choice for your organization, here are a few questions that you need to consider:

1. What is an organization’s goal/outcome? Is it like numbers, functionality, simplification, or just a standardization of business process?

2. Is everything you need inside the system, or will you customize the application to bring avant-garde features?

3. What is your level of expertise and who are your experts? In-house professionals or a Salesforce certified partner?

4. What are the timelines and budget of your Salesforce implementation?

5. Are the end-users going to like it? Think of it from a user’s perspective.

To help you along, here are some answers that you may be seeking:

Business needs, available expertise and desired outcome after implementation

An organization needs to be clear about what it is trying to achieve while implementing Salesforce. A document on what your organization is looking at will help prioritize your business needs and fast-track the desired outcome in numbers or another such element.

An organization needs to have a quick check on the available Salesforce expertise, coding capabilities, core competencies and skills. It may be an in-house team or a preferred Salesforce partner to help. Will the team manage and support Salesforce implementation throughout, or will they need assistance from a Salesforce partner?

Pros and cons of each method

Whether you opt for a Salesforce configuration or a Salesforce customization, each one comes with its own sets of advantages and challenges. Here is table that will help analyze each method quickly:

Know what’s best for you

Go ahead with Salesforce configuration when the needs can be fulfilled by using the original Salesforce application’s in-built tools, functionalities, and modules. You may also choose this method when you know you have limited time to implement Salesforce and you do not intend to spend a heavy amount.

Choose Salesforce customization when you are looking at a solution that needs to specify your user’s customization of certain features functions and integrations of third-party tools. Also, keep an eye on having niche expertise, budgets, and longer timelines. At times, you might have to choose a mix of both methods.

Your organization’s selection of any method or a mix of both will always depend on your unique needs, goals, and the outcome you seek. The best possible way to approach this would be to appoint Salesforce expert who will properly consult configure, implement Salesforce to its best, run it for some time, and check the results. If you haven’t been able to achieve your desired outcome, perhaps because some features and functionalities are still missing, you can go for Salesforce customization to usher in the missing pieces.

Neurealm is well-equipped to be your Salesforce Partner. Our experts can guide your team to understand what is best suited for your organization’s aspirations and IT goals. Read more about how we have helped other clients maximize Salesforce in their organizations: https://www.gslab.com/partners/salesforce Reach out to us at info@gslab.com to learn more about how we can help you optimize Salesforce for value creation.

Co author/Contributor of the blog : Shruti Vishnupurikar | Senior Marketing Executive

   Author

Animesh Patel | Associate Director Engineering | Enterprise Technologies (Salesforce & MuleSoft) | Neurealm

Animesh Patel serves as Associate Director Engineering at Neurealm and heads Neurealm’s Vadodara development center. He leads engineering and delivery of Enterprise Application Development and Enterprise Integrations related projects using platforms like Salesforce & MuleSoft. He has 14+ years of experience in building scalable software products & solutions in many different domains like, IAM (Identity & Access Management), GRC (Governance, Risk & Compliance), Security, Virtualization, Fitness tracking and CC & UC (Contact Center & Unified Communications) related products. Over the years, he helped building engineering teams in various domains and technologies and successfully delivered multiple customer projects.

The post Salesforce Configuration or Customization appeared first on Neurealm.

]]>
Migration of VM based deployment to Kubernetes https://www.neurealm.com/blogs/migration-of-vm-based-deployment-to-kubernetes/ Wed, 08 May 2024 05:22:49 +0000 https://gavstech.com/?p=14217 The post Migration of VM based deployment to Kubernetes appeared first on Neurealm.

]]>

I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Organizations are quickly migrating from virtual machines (VMs) to a cloud-native platform constructed around container technologies like Kubernetes to deal with high application availability, easier scalability and resilience.
In this blog, we discuss key factors to consider while migrating VM-based applications to Kubernetes. This may be useful for those who are currently running their microservices without any containerization, or have containerized but without any orchestration and want to provide high availability, scalability and failover.

A Deployment Model without K8s

Take a look at the following image — a diagrammatic representation of a deployment model with Kubernetes where microservices are running on the same VM or distributed across multiple VMs.

According to this model, inter-service communication would be possible with the following options:

  • Services point directly to localhost:{{port}}
  • Services point to arbitrary service name, for example, dbhost:5432, and maintain IP address against such service names in /etc/hosts file as shown below,
    #: vi /etc/hosts
    127.0.0.1 dbhost

Now, let’s discuss the challenges with this model.

Scalability

  • It is not possible to run multiple service instances with the same port on the same VM. This restricts scaling up the application whenever it is required to serve large volumes of requests. Hence, horizontal scaling is not feasible.

Load Balancing

  • Multiple service instances may run on different VMs with multiple entries in the hosts file to achieve scalability. Even then, caller service cannot invoke distributed instances at once of such a target service.
  • A dedicated load balancer is required to overcome this. The load balancer will resolve the hostname against the IP address of the VM where the service instance is running.

Failover

  • It is important that application uptime is high. A service may abruptly stop working because it is out of memory or insufficient disk space.
  • In such cases, service should auto-restart or another service instance should spawn.
  • Regular health check probes need to be implemented to detect and take a corrective course of action in such scenarios.

Availability

  • In an instance when the entire VM shuts down abruptly, the whole application is down.
  • Providing application uptime during release cycles is also a challenge. End users may experience downtime when application upgrade is in progress.

Monitoring

  • With self-managed services, it is necessary to implement additional monitoring solutions that would provide visibility of all the services showing their health, memory consumption and other such insights.
Migration to Kubernetes

Docker

(Note: You may skip this section if microservices are already packaged inside docker images.)
A docker image is a lightweight, standalone, executable package of software that includes everything needed to run an application.

To containerize a microservice, here is an overview of the steps,

  • Write Dockerfile that specifies base image, directory structure, application artifacts and an entrypoint script that launches the application.
  • A CI/CD pipeline that builds service, builds a docker image and pushes it to a repository. (The pipeline can also be extended till the actual deployment of the docker image to a target node). Bitbucket pipelines or Jenkins pipelines are possible candidates.

Kubernetes

Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. It can be configured in cluster mode or on a single machine i.e. minikube that is suitable for local testing.

Cluster

Microservices are run inside a Kubernetes cluster. A cluster typically consists of —

  • A master node (VM)
  • One or more worker nodes (VM)

Services are run on worker nodes whereas the master node manages the entire cluster. Outside the cluster, a storage VM contains persistent data. The reason why the storage VM is outside the cluster is to prevent data loss in the event a total cluster failure.
>> You can follow this official documentation to setup a Kubernetes cluster.

Objects

Once the docker image is ready and pushed onto a repository, it is time to add Kubernetes resources for microservices.

Below is a list of the bare minimums required:

  1. Deployment
    • When a docker image is run, a container is created, with its own IP address.
    • On the Kubernetes cluster, a container is run inside a pod, which contains one or more containers and has its own IP address.
    • Deployment is the type of object that creates pods. Pod creation can be autoscaled based on the resource utilization or the static pod count can be set. For a microservice, there will be one deployment that acts like a group of service instances.
  2. StatefulSet
    • There are services that need to persist the data, such as Postgres, RabbitMQ and ElasticSearch.
    • It is advised to use StatefulSet instead of a deployment for such services.
  3. Persistent Volume
    • Persistent data from the containers should be kept on a storage VM or on a cloud service. Persistent Volume object contains configuration of such target locations, along with the storage types and required sizes.
    • Target locations could be NFS or cloud services like Azure Files, Azure Disk, AWS EBS, and AWS S3.
    • Instead of associating with Persistent Volume directly, pods are associated with Persistent Volume Claim, which demands for a certain storage type and size. Whichever Persistent Volume satisfies that criteria, is then linked with Persistent Volume Claim.
    • When mounting directory on storage node, it is important to provide fsGroup under pod specifications as follows,securityContext:
      fsGroup: 2000Note: The “2000” mentioned here is the group permission of the target directory on the storage node.
  4. Service
    • A service object is a key for inter-service communication inside the Kubernetes cluster. For a microservice, there should be a Kubernetes service present.
    • The service name should be the same as the hostname used by the application to call another service.
    • Kubernetes maintains an endpoint mapping of service name vs IP addresses of pods. Here, hostname resolution is done out-of-the-box, eliminating the need to maintain entry in the hosts file.
    • By default, service type is ClusterIP which is not exposed outside the Kubernetes cluster, but a NodePort type of service is. A LoadBalancer service type is applicable in the cloud where a public IP address associated with the cloud load balancer service is assigned to the Kubernetes service.
  5. Ingress
    • Ingress are the rules that redirect certain requests to a specific service based on the request path. These are read by the Ingress Controller, responsible for routing and load balancing requests. For example, NGINX Ingress Controller (ingress-nginx) is maintained by the Kubernetes community and provides yaml directives that can be used while writing ingress rules.
    • Other options for Ingress Controller are HAProxy, NGINX Plus and Azure Gateway Ingress Controller.
    • Ingress Controller acts like a load balancer because it forwards a specific request to the destination pod.
    • Along with ingress rules, it is necessary to install Ingress Controller, which has its own pod and service. The service is of type NodePort. This would be the single entry point to the entire Kubernetes cluster.
  6. Secret
    • Secret is a type of object that maintains data in the form of key and value, where values are base64 encoded.
    • It is also important to create a secret that has credentials for pulling docker images from a private repository. The secret name should be then provided against imagePullSecrets in pod specification.
  7. ConfigMap
    • Similar to Secret, ConfigMap can be used to store data in key and value form, which could then be passed to pod as an environment variable. Values are not base64 encoded and stored in plain text form.
  8. Namespace
    • Namespace is the logical separation of Kubernetes objects inside the cluster, for example, a namespace with infrastructure services containing a database server could serve Dev and QA pods from their respective namespaces.
    • If no namespace is provided, then all the resources are deployed in the default namespace.
  9. Deployment Model
    • Although the ingress controller is capable of SSL termination, this could also be delegated to a gateway service running on the master node.
    • Gateway service would conduct an SSL termination and forward requests to the Kubernetes cluster.

Take a look at what a final deployment model would look like:

Notice how the ingress controller is the single entry point to access the cluster. There should be a Kubernetes service and a deployment for each microservice. Ingress rules are implemented by Ingress Controller. Persistent Volume and Persistent Volume Claim can be shared across multiple pods.

Packaging and Deployment
  • Kubernetes resources can be deployed using the following command:
    kubectl apply -f resource.yaml
    where the resource.yaml contains Kubernetes resources.
  • This approach does not involve packaging and release considerations. Helm can be used to package and automate deployment.
  • With Helm, we could define, install, and upgrade Kubernetes resources. Helm charts need to be created that contain Kubernetes resources in a template form, and their values can be injected from external files.
  • On top of Helm, automating helm charts can be done using tools like Helmfile or Helmsman.
Key factors to keep in mind

You may be still contemplating your move to Kubernetes, or may have already started off on your journey. However, it would be incomplete without highlighting the following points:

  • Kubernetes provides support for horizontal scaling of microservices.
  • Load balancing of the requests is taken care of by the ingress controller.
  • With the pod states, Kubernetes is able to spawn additional pods, providing a failover capability.
  • A pod-readiness state can be controlled with a readiness probe, which provides feasibility to upgrade/rollback services without impacting the application uptime.
  • A Kubernetes dashboard provides an out-of-the-box monitoring solution to monitor the entire cluster.
How to maximise Kubernetes for better application infrastructure

Kubernetes is the perfect solution for application scalability, visibility, effort-savings and rollouts, while taking care of the complete application infrastructure and easing the overall development lifecycle. However, the perfect solution needs a worthy system integration partner for a streamlined implementation.

Neurealm boasts of extensive hands-on experience, working with container orchestration tools and our product engineering expertise has helped in more than 2,000 booming product releases.

>> Here’s an example of Neurealm’s successful VM migration for a WAN company where we successfully migrated five clusters in one year, each consisting of more than 25 microservices hosted by more than 50 AWS instances. Click here to read now: https://www.gslab.com/case-studies/migrating-wan-product-from-microservices-to-kubernetes

   Author

Mayur Dhande | Senior Software Engineer

Mayur has 7+ years of experience in Software Development. He is an experienced backend developer and works on technologies in the cloud integration and healthcare domain. Mayur possesses industry experiences in Web Development, Microservices, DevOps and Azure Cloud. His area of interest includes Web development, Automation and Machine Learning.

The post Migration of VM based deployment to Kubernetes appeared first on Neurealm.

]]>
Migration of Standalone Spark on Kubernetes to gain maximum efficiency from containers https://www.neurealm.com/blogs/migration-of-standalone-spark-on-kubernetes-to-gain-maximum-efficiency-from-containers/ Wed, 08 May 2024 05:22:45 +0000 https://gavstech.com/?p=14222 The post Migration of Standalone Spark on Kubernetes to gain maximum efficiency from containers appeared first on Neurealm.

]]>

Apache Spark is a large-scale open-source data processing framework. The standalone Spark cluster consists of a master node and multiple worker nodes. However, as the need to scale up and down becomes crucial with unpredictable load, it is better to migrate standalone spark applications to Kubernetes. With an extensible open-source orchestration platform like Kubernetes, master and worker nodes can be managed easily in isolation with better resource management. Though there are several ways to deploy standalone Spark Cluster, in this blog, we will focus on Kubernetes that allows us to gain maximum efficiency from containers that can run anywhere.

Below is a diagrammatic representation of Analytics system architecture on standalone spark cluster (VM Based architecture)

Technology Used:
  1. Apache Spark 3.1.1
  2. Elasticsearch
  3. RabbitMQ
  4. Python Scripts & Crontab
Here is the data flow:
  • The Business Services and Apps publish Analytics events to a specific queue of RabbitMQ.
  • The Spark application listens to the same queue and dumps them into Elasticsearch.
  • Analyze data jobs are triggered by Crontab every midnight. First, crontab invokes the python script. Then, the python script calculates the current date, last analyze date, computes the analyze range, and accordingly submits the spark jobs.
  • Spark job performs data massaging and aggregation in different time intervals such as hourly, daily, etc. This analyzed data is stored in relevant Elasticsearch indexes.
  • Analyzed data is queried from the dashboard with graphs.
  • The Angular app shows the analyzed data in charts and/or tabular format as needed.
Migration to Spark managed by Kubernetes

Spark can run on clusters managed by Kubernetes. Here are the steps to create and run the applications on Spark managed by Kubernetes.

1.Code Changes:

Ensure that the spark session is closed at the end; otherwise, spark application pods will remain running forever.

session.stop()

Create an application jar that will be used in docker image creation.

2. Spark setup:

Download the spark zip from the official site and set it up. Below steps are used on CentOS 7 to setup Spark.

# create directory
mkdir -p /opt/apache
cd /opt/apache

# download the spark 3.1.1
wget https://mirrors.estointernet.in/apache/spark/spark-3.1.1/spark-3.1.1-bin-hadoop2.7.tgz

# unzip spark and move it to spark directory
tar -xzf spark-3.1.1-bin-hadoop2.7.tgz
mv spark-3.1.1-bin-hadoop2.7/ spark

# initialize SPARK_HOME and path variables
export SPARK_HOME=/opt/apache/spark
export PATH=”$SPARK_HOME/bin:$PATH”

3. Create Docker Image:

Spark (with version 2.3 or higher) ships with a Dockerfile that can be used or customized based on an individual application’s needs. It can be found in the kubernetes/dockerfiles/ directory.
Spark also ships with a bin/docker-image-tool.sh script that can be used to build and publish the Docker images to use with the Kubernetes.

Copy application jar file to $SPARK_HOME/jars/ directory. Only one application jar file allowed in one image. Run below commands to build and push the docker image to the repository.

cd $SPARK_HOME/bin

docker-image-tool.sh -r repo.com/path/spark -t spark-analyze-latest build

docker-image-tool.sh -r repo.com/path/spark -t spark-analyze-latest push

We need to create separate docker images for each application in our case analyze and streaming applications. These images will be used in the /spark-submit command later.

Architecture:
[vc_single_image image=”7036″]

Every component from the VM-based system is now converted into a separate container. Spark pods are created during job execution and terminated upon completion. For example, here is the spark-submit command that is executed from the scheduler container python script.

$SPARK_HOME/bin/spark-submit
–master k8s://https://10.2.1.11:6443
–deploy-mode cluster
–name spark-streaming
–class com.example.analytic.SparkStreaming
–conf spark.executor.instances=1
–conf spark.executor.cores=2
–conf spark.kubernetes.submission.waitAppCompletion=false
–conf spark.kubernetes.namespace=dev
–conf spark.kubernetes.container.image.pullPolicy=Always
–conf spark.kubernetes.container.image.pullSecrets=regcred
–conf spark.kubernetes.container.image=repo.com/path/spark:spark-streaming-latest
–conf spark.kubernetes.driver.volumes.persistentVolumeClaim.exvolume.mount.path=/opt/spark/work-dir
–conf spark.kubernetes.driver.volumes.persistentVolumeClaim.exvolume.options.claimName=spark-streaming-volume-pvc
–conf spark.kubernetes.file.upload.path=/opt/apache/spark/upload-temp
local:///opt/spark/jars/analyze-assembly-1.0.0.jar ‘2021-05-14 00:00:00’ ‘2021-05-16 00:00:00’

Scheduler container should have permission to create, delete, watch, list, get pods, configmaps and services. In addition, it will be responsible for running and monitoring the job’s status and marking the job as completed upon successful completion.

Conclusion:

Migration of standalone apache to Kubernetes is comparatively cost effective and provide dramatic cost benefits with efficient resource sharing. Neurealm has extensive expertise working with container orchestration tools and our product engineering expertise has helped in 2000+ booming product release.

   Author

Shivaji Mutkule | Lead Software Engineer

Shivaji has 7+ years of experience in Software Development. He is an experienced FullStack Developer and works on cutting-edge technologies in the healthcare domain. Shivaji possesses industry experiences in Web Development, Analytics, Microservices, DevOps, and Azure Cloud Networking. He has completed M.E. in Computer Science and Engineering from Government College of Engineering, Aurangabad(GECA). His area of interest includes Web development, Data Mining, and Machine Learning.

The post Migration of Standalone Spark on Kubernetes to gain maximum efficiency from containers appeared first on Neurealm.

]]>
Requirements Analysis in QA https://www.neurealm.com/blogs/requirements-analysis-in-qa/ Wed, 08 May 2024 05:19:24 +0000 https://gavstech.com/?p=14185 The post Requirements Analysis in QA appeared first on Neurealm.

]]>

Software Requirement is a detailed description of the system under implementation. It outlines the practical usage of a product or Service, Condition, or capability to which a system must conform. Requirements can range from high-level abstract statements of services or system constraints to detailed mathematical functional specifications. Here we will discuss requirement analysis and its consideration w.r.t. QA.

Software development life cycle (SDLC) models describe different phases of the software cycle and the order in which those phases are executed – Requirements gathering and analysis, Design, Implementation or coding, Testing, Deployment, and Maintenance.

What is Requirement Analysis: It is the process of determining user expectations for a system under consideration. These should be quantifiable and detailed.

Requirement Analysis:

  • Serves as a foundation for test plans and project plan
  • Serves as an agreement between developer and customer
  • The process of making stated and unstated requirements clear
  • The process to validate requirement for completeness, unambiguity, and feasibility.

The below picture depicts the consequence of poor requirement analysis and its impact on the Software development life cycle.

A number of devices based on mechanical principles were invented to help in computing leading to even analog computers. The computational theories also evolved with the advent of logarithms etc.

Here we can clearly see if the requirement analysis is not done in early phase of the SDLC then its impact is huge to fix it in later phases. Few consequences of poor requirement analysis are like incorrect feature delivery, poor product quality, number of change control to fix system flaws, extension of project deadlines etc.  More we delay in analysing the requirement more it costs and which impacts project delivery and quality.

Listed challenges in Requirement Analysis phase in QA:

  • In early stage of SDLC the scope is not defined clearly.
  • Many times there is ambiguous understanding of processes.
  • Communication between project team and stakeholders plays important role.
  • Insufficient inputs from customer leads to assumptions and those are not accepted in UAT.
  • Inconsistency within single process in multiple users
  • Conflicting customer views.
  • Frequent new requirements.

Tools and techniques used for analyzing the requirements are,

  1. Use Cases: It is a methodology used in requirement analysis to identify, clarify, and organize the requirements. It is set of possible sequences of interactions between systems and users in a particular environment and related to a particular goal.
  2. Requirement Understanding Document (RUD) – Document covers details of Requirement understanding pertaining below points:
    • Assumptions
    • System Details
    • Logical System Requirements
    • System Entity
    • Hardware
    • Acceptance Criteria
  3. Prioritize each requirement
  4. Discuss with team and identify testing scope
  5. Break down requirements in tasks and user stories.

How to Analyse Requirements?

  • Find out what software has to do.
  • Identify requirements by questioning like, Why, What, Who, How etc.
  • Find out how complex application would be and its impact on testing.
  • Which all things would need to be tested.

Requirement validation: Validate requirements based on below points so that at the end of the requirement analysis phase all required information available.

  1. Correctness: find out incorrect statement/requirement.
  2. Completeness: find missing requirement.
  3. Feasibility: find what all features are possible to test and which are beyond the scope.
  4. Testability: Different testing applicable.
  5. Ambiguity: find single interpretation of requirements (statement not clear due to multiple meanings).
  6. Consistency: find out requirement consistency and pointing to single requirement.

After validating whole requirement go ahead and categorize it into 3 types, functional, non-functional and special requirements. This categorization will help in creating detailed Test cases for different testing types.

QA Role:

QA involved in requirement analysis activity to ensure that the requirements identified by the BA and accepted by the customer are measurable. Also this activity provides inputs to various stages of SDLC to identify the availability of resources, scheduling and testing preparation. Below activities QA need to perform.

  • Analyze each and every requirement from specification document, use cases.
  • List down high level scenarios.
  • Clarify queries and functionality from stakeholders.
  • Promote suggestions to implement the features or any logical issues.
  • Raise defect or clarification against the specification document.
  • Track the defect or clarification rose against the specification document.
  • Create high level Test Scenarios.
  • Create Traceability Matrix.

Outcome of the Requirement Analysis Phase:

  • Requirement Understanding Document.
  • High level scenarios.
  • High level test strategy and testing applicability.

With all above mentioned techniques and checklist of requirement analysis, Tester is ready Sign off from Requirement Analysis phase.

   Author

Sarika Gadmade

The post Requirements Analysis in QA appeared first on Neurealm.

]]>