bomonike

microsoft-ai.png How to automate AI workflows in Microsoft’s Azure and Fabric, despite marketing rebrands, and passing AI-900 & AI-102 certification exams.

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Here is a hands-on deep dive introduction through all the information I’ve found around Microsoft’s Artificial Intelligence (AI) offerings running on the Azure cloud.

My contribution to the world (to you) is a less overwhelming learning sequence, one that starts with the least complex of technologies used, then more complex ones. NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Acronyms & Glossary

Machine Learning vs. AI services vs. LLMs vs. Generative AI

Join Communities

AI is moving fast, so everything breaks. Like walking across a desert, we don’t get very far on our own.

  1. Sign up for https://discord.com/invite/6GUBsZfMBq

  2. Sign up for Microsoft for Startups Founders Hub to receive free OpenAI credits and up to $150k towards Azure credits to access OpenAI models through Azure OpenAI Services.

  3. Review issues in the GitHub at

    https://github.com/microsoft/generative-ai-for-beginners/issues

  4. If you find something wrong in the content, add an issue in its GitHub.

Prerequisites to ai.azure.com (AI Foundry)

Many give up entirely because they did not get their working environment properly setup before diving into the “For Beginners” courses.

This is covered in <LEARN modules certification AZ-104:

Using Command Line:

  1. Install fancy Terminal program Warp.

  2. Install package manager Homebrew (if you’re on macOS).

  3. Install Git.
  4. Install Visual Studio Code and fancy IDEs Windsurf, Cursor, Claude, etc.
  5. Install Python interpreter.
  6. Install Conda.
  7. Install JupyterLab.
  8. Install Azure CLI.
  9. Install Microsoft Edge browser.

  10. Install Google’s Chrome browser.

    On Microsoft's Edge browser:

  11. Create an email (on protonmail.com or gmail.com)
  12. Save the password in a password manager.
  13. Credit card. If you have a Capital One card, generate a unique card number.
  14. Put the expiration date as an event in your calendar.

  15. Phone. If you can, generate a unique phone “burner” number.

  16. Get onboarded to a Microsoft Azure subscriptions and learn Portal GUI menu keyboard shortcuts on the Azure portal at

    https://portal.azure.com/?quickstart=True#view/Microsoft_Azure_Resources/QuickstartCenterBlade

    https://portal.azure.com (Azure Cloud Shell)

  17. Setup a CLI scripting environment in https://shell.azure.com like I describe in my mac-setup page

    WARNING: A storage account is required for the CLI to work AND it costs money per month.

  18. Open your Command Terminal. On macOS, use Terminal. On Windows, use cmd, Bash (Windows Terminal) or PowerShell.

  19. Use CLI to Create a Cognitive Service to get keys to call the first REST API from among sample calls to many REST APIs: the Translator Text API.

  20. On Windows 11, install Edge browser according to

    https://microsoftlearning.github.io/mslearn-ai-services/Instructions/setup.html

    The setup is for this LAB pop-up:

    https://learn.microsoft.com/en-us/training/modules/create-manage-ai-services/5a-exercise-ai-services

  21. Setup PowerShell scripts

  22. Create Azure master account using email, credit card, and phone.
  23. Create Subscription with credit card.
  24. Azure AI Foundry Account
  25. Create Azure Project (with Hub)

  26. View Hub in the Azure AI Foundry Management center

  27. Define IAM Roles and accounts for Azure AI Foundary
  28. Define Quota

  29. View Azure Cost analysis

  30. Create a GitHub account using your email address.

  31. Create a GitHub Personal Access Token to use Github Models Marketplace free access to LLMs used to create AI Agents.

  32. Open Command Prompt or Terminal.
  33. Create a folder for your GitHub account.

  34. Navigate to the folder for your GitHub account.

  35. Go to your Personal Access Tokens settings in your GitHub Account.
  36. Select the Fine-grained tokens option on the left side of your screen.
  37. Select Generate new token.
  38. Copy your new token that you have just created.
  39. Pick a region.

  40. Add token & region to the .env file for this course.

  41. In the https://portal.azure.com
  42. In the https://ai.azure.com/ “Azure AI Foundry Portal”
  43. In the https://ml.azure.com/ “Azure Machine Learning Portal”

Beginners Series from Microsoft Cloud Advocates

DO THIS: Click “Star” and “Watch” for “All Activity” in each repo below:

How to Make Money from this?

End-user Productivity of Human-Computer Interaction (HCI)

Microsoft “democratizes” Machine Learning and AI by providing a front-end GUI that hides some of the complexities, enabling them to be run possibly without programming.

  1. DEMO: Hands-on with AI/Guidelines for Human-AI Interaction: Click each card to see examples of each guideline

    https://aka.ms/hci-demo which redirects you to
    https://aidemos.microsoft.com/guidelines-for-human-ai-interaction/demo

    • Initially - make clear what the system can do & how well the system can do what it can do.
    • During interaction - Time services based on context; show contexually revelvant info; Match revelvant social norms; Migrate social biases.
    • When Wrong - support efficient invocation, dismissal, correction; Scope services when in doubt.
    • Over Time - remember recent interactions; learn from user behavior; update and adapt cautiously; encourage granular feedback

    PROTIP: Some fonts are real small. Zoom in to read it.

    Hybrid workflow Pipelines

    PROTIP: Although most of Microsoft’s product documents focus on one technology at a time, actual production work enjoyed by real end-users usually involves a pipeline consisting of several services. For example: ingesting (stream processing) a newsfeed NLP (Natural Langague Processing):

    az-ml-newsfeed-546x623

    That and other flows are in the Azure Architecture Center.

  2. For more about ALGORITHMS USED, see my explanations at https://bomonike.github.io/machine-learning-algorithms, which lists them by alphabetical order and grouped by.

Case studies of how people are already making use of AI/ML to save time and money:

Microsoft’s AI products portfolio

microsoft-ai-portfolio-1725x989.png
Click picture for full-page view.

In the diagram above, Microsoft makes a distinction between “Business Users & Citizen Developers” who use their Applications and “Power Platform” and geeky “Developers & Data Scientists” who use “Azure AI” in the Azure cloud.

In the diagram above, Microsoft categorize Azure’s AI services these groups (all of which have GUI, CLI, and API interfaces):

PROTIP: Several services are NOT shown in the diagram above. The list in a Microsoft LEARN module show a different order:

ms-ai-def-1187x508.png

DEFINITION: There are what are called “narrow” or “weak” AI.

Azure Cloud Services

  1. Click (Azure cloud) SERVICES USED:

    • Azure Machine Learning
    • PowerBI
    • Azure Blob Storage
    • Azure HDInsight
    • Azure Data Factory
    • Azure SQL DataWarehouse
    • Azure App Service
    • Azure Sql
    • Azure Event Hubs
    • Azure Stream Analytics
    • Azure Cognitive Service - LUIS
    • Azure Data Lake Analytics
    • Azure Data Lake Store
    • Azure Virtual Machine
    • Azure Batch

Availability of AI Products

Every product Microsoft offers is listed, by region in this table showing availability (regardless of whether you have permissions to each specific product).

microsoft-ai-avail-240108-2544x1796.png

Cognitive Services List

The method shown in this section gives you a list of all “kinds” of services Azure currently provides for your Azure subscription.

After your Command Terminal environment on your laptop has the Installed prerequisites of obtaining an Azure account with subscription and installed the “az” CLI modules needed to run “az” commands on your Terminal, run this command:

PROTIP: “ContentModerator” has been deprecated.

Cognitive Services Grouping

PROTIP: The services above are listed in random order. But the table below groups services to help you quickly get to links about features, tutorials, and SDK/API references quickly.

As of this writing, in various marketing and certification training DOCS, Azure Cognitive Services are grouped into these (which is the basis this article is arranged. Click on the underlined and bolded category name to jump to the list of services associated with it, in this order (like on the AI Products Portfolio diagram:

  1. AI Vision (Visual Perception) provides the ability to use computer vision capabilities to accept, interpret, and process input from images, video streams, and live cameras. Interpret the world visually through cameras, videos, images

  2. AI Speech - Text-to-Speech and Speech-to-Text to interpret written or spoken language, and respond in kind. This provides the ability to recognize speech as input and synthesize spoken output. The combination of speech capabilities together with the ability to apply NLP analysis of text enables a form of human-compute interaction that’s become known as conversational AI, in which users can interact with AI agents (usually referred to as bots) in much the same way they would with another human.

  3. AI Language - aka Natural language Processing (NLP) to translate text (Text Analysis), etc.

  4. AI Decision (Making) provides the ability to use past experience and learned correlations to assess situations and take appropriate actions. For example, recognizing anomalies in sensor readings and taking automated action to prevent failure or system damage. supervised and unsupervised machine learning

  5. Other: OpenAI (to power your apps with large-scale AI models) is a recent add to this confusing category because of so many branding changes (Cortana, Bing, Cognitive, OpenAI, etc.)

Text analysis and conversion provides the ability to use natural language processing (NLP) to not only “read”, but also generate realistic responses and extract semantic meaning from text.

Pricing

PROTIP: The table below groups each kind of cognitive AI service along with how many FREE transactions Microsoft provides for each service on its Cognitive Services pricing page.

IMPORTANT PROTIP: Microsoft allows its free “F0” tier to be applied to only a single Cognitive Service at a time. To remain free, you would need to rebuild a new Cognitive Service with a different “Kind” between steps.

Pricing of AI Services Kinds

Within each grouping, each service is listed in the sequence within that group’s LEARN module.

Demo Kind FREE API
global "CognitiveServices" See below -
? "AIServices" ?
? "Internal.AllInOne" ?
AI Vision
MSR "ComputerVision" => AI Vision 2 TPC, 10K/mo. API
MSR "CustomVision.Prediction" 10K predictions/mo. API
Features "CustomVision.Training" 2 TPS, 1 hour/mo. API
MSR "Face" => AI Vision 20 TPM, 30K/mo. API
Features Video Indexer (Applied AI) 600 minutes in portal, 2400 in API API
- Ink Recognizer" (DELISTED) - -
AI Speech
MSR "SpeechServices" => AI Speech (Speech-To-Text", Text-to-Speech") 1 concurrent, 5 hours/mo.* API
Features "SpeakerRecognition" (DELISTED?) 20/min, 5K/mo. API
Features Speech Translation (DELISTED?) 20/min, 5K/mo. API
MSR AI Language
MSR "TextTranslation" => AI Translator 2M chars/mo. API
Features "LUIS" (Language Understanding, UNLISTED) 5 TPS, 10K trans/mo. API
Features "LUIS.Authoring" 5 TPS API
Features "QnAMaker" (UNLISTED) 3 TPS, 50K trans/mo, 3 docs/mo. API
Features "QnAMaker.v2" 3 TPS, 5K/mo. API
Features "TextAnalytics" (Detect Language, Sentiment, Named Entities, Key Phrases) 5K trans/mo. API
Features "ImmersiveReader" (Applied AI) 3M chars/mo. API
? "LanguageAuthoring" ?
AI Decision (Making)
Features "AnomalyDetector" 2,000 trans/mo. API
Features "ContentModerator" (deprecated) 1 TPS API
Features "ContentSafety" ?
Features "MetricsAdvisor" (Applied AI) preview API
Features "Personalizer" 50K trans/mo. API
Search
? KnowledgeMining (UNLISTED) ?
? Microsoft Bing (Applications) ?
Features "Bing.CustomSearch" (UNLISTED) 50K trans/mo. -
Features "Bing.Search.v7" (UNLISTED) 3 TPS, 1000/mo API
? AI Search Cognitive Search (Applied AI) ?
MSR Document Intelligence = FormRecognizer (Applied AI) 500 pages/mo. API
Other
? "HealthInsights" ? ?
$ Bot Service (Applied AI) ? ?
Features OpenAI Service ? SDK

Links are provided for each service to its Features and API/SDK pages.

MSR CognitiveAllInOne GUI Responsible AI

“MSR” in the table above identifies a Multi-Service Resource accessed using a single key and endpoint to consolidate billing.

PROTIP: HANDS-ON: Create the CognitiveAllInOne resource using the GUI so that you can acknowledge Microsoft’s terms for Responsible AI use:

microsoft-ai-resp-288x63.png

PROTIP: Apply of Terraform to create AI service resources will error out unless that is checked.

Microsoft’s Confusing Branding

Microsoft’s AI offerings have gone through quite a bit of churn.

Microsoft has invented several names to refer to their AI offerings:

Cortana => Bing => Cognitive Services => OpenAI => Generative AI => AI Language

“Cortana” was the brand-name for Microsoft’s AI. Cortana is the name of the fictional artificially intelligent character in the Halo video game series. Cortana was going to be Microsoft’s answer to Alexa, Siri, Hey Google, and other AI-powered personal assistants which respond to voice commands controlling skills that turn lights on and off, etc. However, since 2019, Cortana is considered a “skill” (app) that Amazon’s Alexa and Google Assistant can call, working across multiple platforms.

For Search, the Bing “Bing” brand, before OpenAI was separated out from “Cognitive Services” to its own at https://docs.microsoft.com/en-us/azure/search, although it’s used in “Conversational AI” using an “agent” (Azure Bot Service) to participate in (natural) conversations. BTW: in 2019 Cortana decoupled from Windows 10 search.

Since October 31st, 2020, Bing Search APIs transitioned from Azure Cognitive Services Platform to Azure Marketplace. The Bing Search v7 API subscription covers several Bing Search services (Bing Image Search, Bing News Search, Bing Video Search, Bing Visual Search, and Bing Web Search),

Azure IoT (Edge) Services are separate.

The 2023 rebranding for Microsoft’s AI services to mimic human intelligence is “AI Language”, which includes Cognitive Services and Bing.

PROTIP: On Jan 8, 2024, https://aka.ms/language-studio had “coming soon” for Video and Learn, and “preview” for several services. Essentially Microsoft has two separate offerings by different groups:

Microsoft has three service “Providers”:

Asset type Resource provider namespace/Entity Abbre- viation
Azure Cognitive Services Microsoft.CognitiveServices/accounts cog-
Azure Machine Learning workspace Microsoft.MachineLearningServices/workspaces mlw-
Azure Cognitive Search Microsoft.Search/searchServices srch-

Microsoft has been the “sole provider” of servers to OpenAI as part of some agreement that counts as investment.

In 2025, Microsoft was not part of the “$500 billion investment” announced in the Trump White House.

A few weeks later, Microsoft announced “Microsoft AI” (MAI).

Microsoft’s History with AI

In April 2018 Microsoft reorganized into two divisions to offers AI:

At Build 2018, Microsoft announced Project Brainwave to run Google’s Tensorflow AI code and Facebook’s Caffe2, plus Microsoft’s own “Cognitive Toolkit” (CNTK).

Microsoft has advanced hardware:

Microsoft Conversational AI Platform for Developers is a 2021 book published by Apress by Stephan Bisser of Siili Solutions in Finland. The book covers Microsoft’s Bot Framework, LUIS, QnA Maker, and Azure Cognitive Services. https://github.com/orgs/BotBuilderCommunity/dashboard

Microsoft Mechanics YouTube channel is focused on Microsoft’s AI work.

VIDEO: What runs ChatGPT? Inside Microsoft’s AI supercomputer

NOTE: Wikipedia has 6,781,394 articles containing 4.5 billion words. English Wiktionary contains 1,439,188 definitions. Webster’s Third New International Dictionary has 470,000 English words.

https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/

Microsoft Reactor ran a “Skills Challenge” to reward a badge for those who complete an AI tutorial.

https://learn.microsoft.com/en-us/training/challenges

https://www.youtube.com/watch?v=ss-kyogPRNo by Carlotta

Microsoft competes for talent with Google, Amazon, IBM, China’s Tencent, and many start-ups.

Azure AI certifications

#AI-900 Azure AI Fundamentals AI-102 Azure AI Engineer Associate
-ml.azure.com Azure Machine Learning Foundry Portal Exercises for MS LEARN FAW DOCS ai.azure.com Azure AI Foundry Portal FAQ DOCS
1.AI Overview
3 hr 2 min		 Exercises: Get started with Azure AI Services
5 hr 5 min
2.Computer Vision
1 hr 40 min		 Exercises: Create computer vision solutions with Azure AI Vision
5 hr 1 min
3.Natural Language Processing
2 hr 39 min		 Exercises: Develop natural language processing solutions with Azure AI Services
7 hr 4 min
4. Document Intelligence and Knowledge Mining
1 hr 19 min		 Exercises: Develop solutions with Azure AI Document Intelligence
2 hr 3 min
Exercises: Implement knowledge mining with Azure AI Search
6 hr 24 min
5.Generative AI
3 hr 32 min		 Exercises: Develop Generative AI solutions with Azure OpenAI Services
2 hr 13 min

Among Microsoft’s Azure professional certifications illustrated by this pdf, there are three levels of AI:

  1. AI-900 $99 Fundamentals is the entry-level exam. It’s a pre-requisite for:

  2. AI-102 $165 Associate exam focuses on the use of pre-packaged cloud-based services for AI development. It has free re-cert after 1-year.

    https://www.linkedin.com/in/alison-felix/ notice that in exercises some of the items are duplicated. For example:

    • The 1st one goes to …Exercises/ai-foundry/01-azure-search.html
    • The 2nd one goes to …Exercises/01-azure-search.html

    Azure AI Foundry FAQ at:

    Azure AI Foundry documentation is at:

    Get 50% off the AI-102 if you finish Coursera’s Microsoft AI & ML Engineering Professional Certificate by Mark DiMauro at Univ. Pittsbergh. It consists of 5 courses:

    1. Foundations of AI and Machine Learning - 35 hours
    2. AI and Machine Learning Algorithms and Techniques - 45 hours
    3. Building Intelligent Troubleshooting Agents - 45 hours
    4. Microsoft Azure for AI and Machine Learning - 21 hours
    5. Advanced AI and Machine Learning Techniques and Capstone - 32 hours

    There is also a Coursera course on “Developing AI Applications with Azure”.

MS Reactor Tutorials

  1. Microsoft Developer Community Blog at https://techcommunity.microsoft.com/category/azure/blog/azuredevcommunityblog covers ALL the various tech Microsoft has (365, Bicep, etc. in Spanish, English, etc.)

  2. https://devpost.com/software/avoid-maga-word-bans/joins/mG6Kzv8uN6hYCjW0jCt9lA

  3. https://devpost.com/submit-to/23928-azure-ai-developer-hackathon/manage/submissions/630940-avoid-word-bans/project_details/edit DevPost Hackthon project described as “Avoid word bans”.

  4. https://www.linkedin.com/pulse/program-find-words-banned-wilson-mar-msc-2ccbc/?trackingId=Ux0g5xgWdyTMf7EAC4hFow%3D%3D

  5. Pamela Fox (MS Evangelist) (pamelafox.org) holds Office hours on Discord

  6. She held in 2025 Microsoft Reactor livestream events in London, etc. on Intelligent Applications https://developer.microsoft.com/en-us/reactor/series/s-1491/

From aka.ms/PythonAI/series to https://github.com/pamelafox/python-openai-demos/discussions/21

  1. https://github.com/pamelafox/python-openai-demos
  2. “Code” button to create GitHub Codespace.

  3. AI Toolkit accesses FREE GitHub Models (these have limited tokens) at https://github.com/marketplace/models

  4. https://gateway.on24.com/wcc/eh/4304051/category/142480/python-ai Python & AI resources

  5. https://github.com/marketplace?type=models

  6. https://github.com/Azure-Samples/llama-index-python This sample shows how to quickly get started with LlamaIndex.ai on Azure🚀

History of AI exams:

George Chen’s Journey to Microsoft Certified: AI Engineer Associate. Videos with PPT file!

AI-900

PROTIP: http://aka.ms/AIFunPath which expands to https://github.com/MicrosoftLearning/mslearn-ai-fundamentals

Exam definitions are at Microsoft’s LEARN</a> includes a free text-based tutorial called “Learning Paths” to learn skills:

Tim Warner has created several video courses on AI-900 and AI-100:

CloudAcademy’s 4h AI-900 video course includes lab time (1-2 hours at a time).

Emilio Melo on Linkedin Learning

Practice tests:

AI-102

AI-102 is intended for software developers wanting to build AI-infused applications.

https://learn.microsoft.com/en-us/credentials/certifications/azure-ai-engineer

First setup development environments:

  1. Read https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy
  2. Get an organizational account (not gmail).
  3. Provide that business address at aka.ms/oaiapply request access to Azure OpenAI service.

AI-102 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

PROTIP: Unlike the AI-100 (which uses Python Notebooks), hands-on exercises at https://microsoftlearning.github.io/AI-102-AIEngineer/ in Microsoft’s 5-day live course AI-102T00: Designing and Implementing a Microsoft Azure AI Solution (with cloud time) consists of C# and Python programs at https://github.com/MicrosoftLearning/AI-102-AIEngineer (by Graeme Malcolm) was archived on Dec 23, 2023 after its content was distributed among these repos:

  1. AI Vision Tutorials
  2. AI Language Tutorials
  3. AI Language
  4. AI Decision Making
  5. Other: OpenAI Tutorials

VIDEO Andrew Brown’s Azure AI Engineer Associate Certification (AI-102) – Full Course to PASS the Exam on FreeCodeCamp’s YouTube channel

https://learn.microsoft.com/en-us/training/courses/ai-102t00 modules:

On Coursera: Coursera video course: Developing AI Applications on Azure by Ronald J. Daskevich at LearnQuest is structured for 5 weeks. Coursera’s videos shows the text at each point of its videos. NOTE: It still sends people to https://notebooks.azure.com and covers Microsoft’s TDSP (Team Data Science Process) VIDEO:

  1. Business understanding (Charter, Objectives, Data sources, Data dictionaries)
  2. Data acquisition and understanding (Clean dataset, pipeline)
  3. Modeling
  4. Deployment model for use (testing) Flow: entry script to accept requests and score them, Deployment configs
  5. Customer Acceptance

Resources:

AI-100 (RETIRED)

On June 30, 2021 Microsft retired the AI-100 exam in favor of AI-102 exam (avilable in $99 beta since Feb 2021). AI-100 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

https://github.com/MicrosoftLearning/AI-100-Design-Implement-Azure-AISol

https://github.com/MicrosoftLearning/Principles-of-Machine-Learning-Python

Resources:

Guy Hummel’s CloudAcademy.com 7hr AI-100 video course.

McK Udemy by Anand Rao Nednur

Raza Salehi created on Pluralsight.com a series for Microsoft Azure AI Engineer (AI-100)

Practice tests:

AI Capabilities

https://learn.microsoft.com/en-us/training/modules/prepare-azure-ai-development/

most medium to large-scale development scenarios it’s better to provision Azure AI services resources as part of an Azure Foundry hub - enabling you to centralize access control and cost management, and making it easier to manage shared resource usage based on AI development projects. Hubs provide a top-level container for managing shared resources, data, connections and security configuration for AI application development. A hub can support multiple projects, in which developers collaborate on building a specific solution. A hub provides a centrally managed collection of shared resources and management configuration for AI solution development. You need at least one hub to use all of the solution development features and capabilities of AI Foundry.

https://learn.microsoft.com/en-us/training/modules/prepare-azure-ai-development/4-azure-ai-foundry

Learning Sequence

This document covers:

  1. Automatically shut down Resource Groups of a Subscription by creating a Logic App.

  2. Run an API connecting to an established endpoint (SaaS) you don’t need to setup: Bing Search.

  3. Create Functions

  4. Create a Workspace resource to run …

  5. Create a Workspace resource and
    Create Compute instance to run
    Automated ML of regression of bike-rentals.
  6. Create Compute instance to run a iPython notebook
  7. Create ML Workspace in Portal, then ml.azure.com

  8. Us cognitivevision.com to Create Custom Vision for …

  9. QnA Maker Conversational AI
  10. Train a Machine Learning model using iPython notebook
  11. IoT - “Hey Google, ask Azure to shut down all my compute instances”.

Automation necessary for PaaS

IMPORTANT PROTIP: As of this writing, Microsoft Azure does NOT have a full SaaS offering for every AI/ML service. You are required to create your own computer instances, and thus manage machine sizes (which is a hassle). Resources you create continue to cost money until you shut them down.

So after learning to set up the first compute service, we need to cover automation to shut them all down while you sleep.

So that you’re not tediously recreating everything everyday, this tutorial focuses on automation scripts (CLI Bash and PowerShell scripts) to create compute instances, publish results, then shut itself down. Each report run overwrites files from the previous run so you’re not constantly piling up storage costs.

Another reason for being able to rebuild is that you if you find that the pricing tier chosen is no longer suitable for your solution, you must create a new Azure Cognitive Search resource and recreate all indexes and objects.

When you use my Automation scripts at https://github.com/wilsonmar/azure-quickly/ to create resources the way you like, using “Infrastructure as Code”, so you can throw away any Subscription and begin anew quickly.

My scripts also makes use of a more secure way to store secrets than inserting them in code that can be checked back into GitHub.

Effective deletion hygiene is also good to see how your instances behave when it takes advantage of cheaper spot instances which can disappear at any time. This can also be used for “chaos engineering” efforts.

To verify resource status and to discuss with others, you still need skill at clicking through the Portal.azure.com, ML.azure.com, etc.

References:

https://github.com/Azure/azure-sdk-for-python

Challenges

Get 50% off by completing just one of these challenges:

Document Intelligence Knowledge Mining

  1. View the marketing page at:

    https://documentintelligence.ai.azure.com/studio

    Azure AI Document Intelligence model overview

  2. Prebuilt models expect a common type of form or document:

    NOT:

    • Resumes
  3. CODING: 
    poller = document_analysis_client.begin_analyze_document(
      "prebuilt-layout", AnalyzeDocumentRequest(url_source=docUrl
   ))
result: AnalyzeResult = poller.result()

  4. LAB: Exercise - Analyze a document using Azure AI Document Intelligence

  5. You are using the prebuilt layout model to analyze a document with many checkboxes. You want to find out whether each box is checked or empty. What object should you use in the returned JSON code? Selection marks record checkboxes and radio buttons and include whether they’re selected or not. NOT Bounding boxes. NOT Confidence indicators.

    PROTIP: Although Word documents are also from Microsoft, DOCX format files are NOT supported by Azure AI Document Intelligence. But PDF documents are supported. Azure AI Document Intelligence is designed to analyze scanned and photographed paper documents, not documents that are already in a digital format so you should consider using another technology to extract the data in Word documents.

  6. Get into the Azure Marketplace to look for Azure AI Document Intelligence – an Azure service that you can use to analyze forms completed by your customers, partners, employers, or others and extract the data that they contain.

  7. https://learn.microsoft.com/en-us/training/modules/plan-form-recognizer-solution/2-understand

References:

AI Landing Zones (ALZ)

In production usage, several Azure subscriptions are needed.

ms-ai-alz-1327x792.png

The above image replaced the previous version) is the Azure Landing Zones OpenAI Reference Architecture defining how to envelope OpenAI with utilities to ensure a defensive security posture. It maps how resources are integrated in a structured, consistent manner, plus ensuring governance, compliance, and security.

The diagram above is an adaptation of Microsoft’s enterprise-scale Azure Landing Zone, a part of Microsoft’s Cloud Adoption Framework (CAF).

Each Design Area has:
ms-az-240217-1238x230.png

A. Enterprise enrollment [TF]
B. Identity and accessment [TF]
C. Management group and subscription organization
D. Management subscription
E. Connectivity subscription
F. AI Services (Landing Zone) subscription
G. Monitoring?
H. Sandbox subscription
I. Platform DevOps Team [TF]</br />

Design Areas F (the Landing Zone for AI):
ms-ai-arch-240217-1268x1318.png

To create the resources in the diagram:
If you prefer using Bicep:

  1. https://github.com/Azure/ALZ-Bicep/wiki/DeploymentFlow
  2. https://github.com/Azure/ALZ-Bicep/wiki/ConsumerGuide
  3. https://learn.microsoft.com/en-us/azure/architecture/landing-zones/bicep/landing-zone-bicep
  4. https://github.com/Azure/ALZ-Bicep/wiki/Accelerator

If you prefer using Terraform:

Included are Private Endpoints, Network Security Groups and Web Application Firewalls.

Sample ML Code

PROTIP: AI-102 is heavy on questions about coding.

Samples (unlike examples) are a more complete, best-practices solution for each of the snippets.

PROTIP: github.com/Azure-Samples from Microsoft offers samples code to use Cognitive Services REST API by each language:

https://docs.microsoft.com/en-us/samples/azure-samples/azure-sdk-for-go-samples/azure-sdk-for-go-samples/

A complete sample app is Microsoft’ Northwinds Traders consumer ecommerce store: install

IMPORTANT: Cognitive Services SDK Samples for:

Tim Warner’s https://github.com/timothywarner/ai100 includes Powershell scripts:

Among Azure Machine Learning examples is a CLI at https://github.com/Azure/azureml-examples/tree/main/cli

PROTIP: CAUTION: Each service has a different maturity level in its documentation at azure.microsoft.com/en-us/downloads, such as SDK for Python open-sourced at github.com/azure/azure-sdk-for-python, described at docs.microsoft.com/en-us/azure/developer/python.

AI Services

The hands-on steps below enables you to operate offline on a macOS laptop to run the Azure AI SDK in a Docker container using VS Code Dev Containers.

https://learn.microsoft.com/en-us/azure/ai-services/

runs from VSCode to run Python 3.10 with Docker containers within Azure AI Studio at https://ai.azure.com

Following How to start:

Work with Azure AI projects in VS Code

  1. Install Docker Desktop, https://code.visualstudio.com/docs/devcontainers/tutorial

  2. Install Visual Studio Code (VSCode). Within VSCode, install Dev Containers extension to use VS Code Dev Container

  3. run the command

    Dev Containers: Try a Dev Container Sample…

  4. Install Git and configure it.

  5. Create a folder “ai_env” from GitHub for the “Sample quickstart repo for getting started building an enterprise chat copilot in Azure AI Studio”:

    FOLDER="ai_env"
git clone https://github.com/azure/aistudio-copilot-sample "${FOLDER}" --depth 1
cd "${FOLDER}"

    Response:

    Cloning into 'ai_env'...
remote: Enumerating objects: 122, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 122 (delta 11), reused 88 (delta 8), pack-reused 0
Receiving objects: 100% (122/122), 227.58 KiB | 951.00 KiB/s, done.
Resolving deltas: 100% (11/11), done.

  6. Define provenance:

    git remote add upstream https://github.com/azure/aistudio-copilot-sample
git remote -v

  7. Open with VSCode to an error:

    code ai_env

  8. Select the “Reopen in Dev Containers” button. If it doesn’t appear, open the command palette (Ctrl+Shift+P on Windows and Linux, Cmd+Shift+P on Mac) and run the Dev Containers: Reopen in Container command.

  9. Install Homebrew

  10. Install the Azure CLI using Homebrew (requires python@3.11?):

    brew install azure-cli

  11. Install the .NET (dotnet) CLI commands:

    DOTNET_ROOT="/opt/homebrew/Cellar/dotnet/7.0.100/libexec"

  12. Add $HOME/.dotnet/tools folder to .zshrc file run at bootup.

  13. Intall miniconda3.

  14. From any folder, create a Conda environment containing the Azure AI SDK:

    conda create --name ai_env python=3.10 pip
yes | conda activate ai_env

    Sample response:

    Collecting package metadata (current_repodata.json): done
Solving environment: done
 
==> WARNING: A newer version of conda exists. <==
  current version: 23.5.0
  latest version: 24.1.1
 
Please update conda by running
 
 $ conda update -n base -c conda-forge conda
 
Or to minimize the number of packages updated during conda update use
 
  conda install conda=24.1.1
 
## Package Plan ##
 
  environment location: /Users/wilsonmar/miniconda3/envs/ai_env
 
  added / updated specs:
 - python=3.10
 
The following packages will be downloaded:
 
 package                    |            build
 ---------------------------|-----------------
 python-3.10.13             |h00d2728_1_cpython        12.4 MB  conda-forge
 setuptools-69.1.0          |     pyhd8ed1ab_0         460 KB  conda-forge
 ------------------------------------------------------------
                                        Total:        12.9 MB
 
The following NEW packages will be INSTALLED:
 
  bzip2              conda-forge/osx-64::bzip2-1.0.8-h10d778d_5
  ca-certificates    conda-forge/osx-64::ca-certificates-2024.2.2-h8857fd0_0
  libffi             conda-forge/osx-64::libffi-3.4.2-h0d85af4_5
  libsqlite          conda-forge/osx-64::libsqlite-3.45.1-h92b6c6a_0
  libzlib            conda-forge/osx-64::libzlib-1.2.13-h8a1eda9_5
  ncurses            conda-forge/osx-64::ncurses-6.4-h93d8f39_2
  openssl            conda-forge/osx-64::openssl-3.2.1-hd75f5a5_0
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
  python             conda-forge/osx-64::python-3.10.13-h00d2728_1_cpython
  readline           conda-forge/osx-64::readline-8.2-h9e318b2_1
  setuptools         conda-forge/noarch::setuptools-69.1.0-pyhd8ed1ab_0
  tk                 conda-forge/osx-64::tk-8.6.13-h1abcd95_1
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
  wheel              conda-forge/noarch::wheel-0.42.0-pyhd8ed1ab_0
  xz                 conda-forge/osx-64::xz-5.2.6-h775f41a_0
 
Proceed ([y]/n)? _

  15. Install the Azure AI CLI using the .NET (dotnet) CLI command:

    dotnet tool install --global Azure.AI.CLI  --prerelease

    Response:

    You can invoke the tool using the following command: ai
Tool 'azure.ai.cli' (version '1.0.0-preview-20240216.1') was successfully installed.

  16. Obtain Python dependency libraries defined in requirements.txt:

    conda install pip
pip install -r requirements.txt

    Alternately, if there were instead an environment.yml.

  17. Get the ai cli:

    ai

    Results with the prerelease version:

    AI - Azure AI CLI, Version 1.0.0-preview-20240216.1
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.
 
This PUBLIC PREVIEW version may change at any time.
See: https://aka.ms/azure-ai-cli-public-preview
 
___ ____  ___  _____
  / _ /_  / / _ |/_  _/
 / __ |/ /_/ __ |_/ /_
/_/ |_/___/_/ |_/____/
 
USAGE: ai  [...]
 
HELP
 
  ai help
  ai help init
 
COMMANDS
 
  ai init [...]             (see: ai help init)
  ai config [...]           (see: ai help config)
 
  ai dev [...]              (see: ai help dev)
 
  ai chat [...]             (see: ai help chat)
  ai flow [...]             (see: ai help flow)
 
  ai search [...]           (see: ai help search)
  ai speech [...]           (see: ai help speech)
 
  ai service [...]          (see: ai help service)
 
EXAMPLES
 
  ai init
  ai chat --interactive --system @prompt.txt
 
  ai search index update --name MyIndex --files *.md
  ai chat --interactive --system @prompt.txt --index-name MyIndex
 
SEE ALSO
 
  ai help examples
 
  ai help find "prompt"
  ai help find "prompt" --expand
 
  ai help find topics "examples"
  ai help list topics
 
  ai help documentation

  18. Navigate to the folder

  19. Initialize the ai project:

    ai init
  20. Select “Initialize: Existing AI project”.
  21. Select “LAUNCH: ai login (interactive device code)”
You have signed in to the Microsoft Azure Cross-platform Command Line Interface application on your device. You may now close this window.
  1. Highlight and copy the code.

  2. Open an internet browser (Safari) to the page

    https://microsoft.com/devicelogin

  3. Click and paste the code from your laptop Clipboard (to authenticate).
  4. Select your user email.
  5. Click Continue to “Are you trying to sign in to Microsoft Azure CLI? Only continue if you downloaded the app from a store or website that you trust.”

  6. Close the browser tab when you see

    You have signed in to the Microsoft Azure Cross-platform Command Line Interface application on your device. You may now close this window.

  7. Confirm that the subscription shown is the one you want to use.

  8. cd to the folder

    Notice that file src/copilot_aisdk/requirements.txt contains:

    openai
azure-identity
azure-search-documents==11.4.0b6
jinja2
  9. Select “(Create w/ standalone Open AI resource)” rather than integrated
  10. Select RESOURCE GROUP, CREATE RESOURCE GROUP, CREATE AZURE OPENAI RESOURCE
  11. Select a name for “** CREATING **”

  12. For “AZURE OPENAI DEPLOYMENT (CHAT)” select “(Create new)”. FIXME:

    CREATE DEPLOYMENT (CHAT)
Model: *** No deployable models found ***
CANCELED: No deployment selected
  13. Select one or skip:
    • ada-embedding-002
    • gpt-35-turbo-16k for chat,
    • gpt-35-turbo-16k
    • gpt4-32k evaluation

  14. Azure AI search resource

    This generates a config.json file in the root of the repo for the SDK to use when authenticating to Azure AI services.

Alternating:

https://github.com/azure-samples/azureai-samples

  1. Select the Node sample from the list.

    File devcontainer.json is a config file that determines how your dev container gets built and started.

Vision services

Vision Tutorial

https://microsoftlearning.github.io/mslearn-ai-vision/ EXERCISES based on
https://github.com/MicrosoftLearning/mslearn-ai-vision

LLMOps: LLMOps Microsoft Developer videos

Use GUI

aka.ms/cognitivevision resolves to YouTube channel “Microsoft Mechanics” rebranded to include “agents”, such as:

  1. In a web browser, navigate to Vision Studio:

    https://portal.vision.cognitive.azure.com/gallery/featured

  2. Sign in.

  3. Select a Subscription.

    VIDEO: The menu of AI Vision service categories uses Microsoft’s “Florence foundational model” (new in 2023) models trained with an “open world” of billions of images combined with a large language model. That enables the identification of objects and their location in a frame such as this:

    microsoft-ai-frame-915x343.png

  4. Click each, then “Try it out”:

    Optical character recognition (OCR):

    • Extract text from images - Extract printed and handwritten style text from images and documents for supported languages.

      Spatial analysis - Video Retrieval and Summary:
    • Video Retrieval and Summary - Generate a brief summary of the main points shown in video. Locate specific keywords and jump to the relevant section.
    • Count people in an area - Analyze real-time or recorded video to count the number of people in a designated zone in a camera’s field of view.
    • Detect when people cross a line - Analyze real-time streaming video to detect when a person crosses a line in the camera’s field of view.
    • Detect when people enter/exit a zone - Analyze real-time streaming video to detect when a person enters or exits a zone in the camera’s field of view.
    • Monitor social distancing - Analyze real-time streaming video that tracks when people violate a distance rule in the camera’s field of view.

    • Face - Detect faces in an image

      Image analysis - click “Try it out” on each of these:
    • Recognize products on shelves - Identify products on shelves, gaps in product availability, and compliance for planograms.
    • Customize models with images - Create custom image classification and object detection models with images using Vision Studio and Azure ML.
    • Search photos with image retrieval - Retrieve specific moments within your photo album. For example, you can search for: a wedding you attended last summer, your pet, or your favorite city.
    • Add dense captions to images - Generate human-readable captions for all important objects detected in your image.
    • Remove backgrounds from images - Easily remove the background and preserve foreground elements in your image.
    • Add captions to images - Generate a human-readable sentence that describes the content of an image.
    • Detect common objects in images - Recognize the location of objects of interest in an image and assign them a label.
    • Extract common tags from images - Use an AI model to automatically assign one or more labels to an image.
    • Detect sensitive content in images - Detect sensitive content in images so you can moderate their usage in your applications.
    • Create smart-cropped images - Create cropped image thumbnails based on the key areas of a larger image.

  5. Click “I acknowledge that this service will incur usage to my Azure account.”

  6. Select a Region (East US, West Europe, West US, or West US 2)

    The “cog-ms-learn-vision” resource is created for you automatically.

  7. Click “View all resources”
  8. Check “cog-ms-learn-vision” created for you automatically.
  9. Click “Select as default resource”.
  10. Close.

Use REST API

https://github.com/Azure-Samples/cognitive-service-vision-model-customization-python-samples

https://microsoftlearning.github.io/mslearn-ai-fundamentals/Instructions/Labs/05-ocr.html

  1. On the Getting started with Vision landing page, select Optical character recognition, and then the Extract text from images tile.

  2. Under the Try It Out subheading, acknowledge the resource usage policy by reading and checking the box.

  3. download ocr-images.zip by selecting

    https://aka.ms/mslearn-ocr-images

  4. Open the folder.

  5. On the portal, select Browse for a file and navigate to the folder on your computer where you downloaded ocr-images.zip. Select advert.jpg and select Open.

  6. review what is returned:

    In Detected attributes, any text found in the image is organized into a hierarchical structure of regions, lines, and words.

    On the image, the location of text is indicated by a bounding box, as shown here:

App for the blind: VIDEO: INTRO: SeeingAI.com. Permissions for the “See It All” app are for its internal name “Mt Studio Web Prod”.

https://docs.microsoft.com/en-us/learn/paths/explore-computer-vision-microsoft-azure

HISTORY: In 2014, Microsoft showed off its facial recognition capabilities with a website (how-old.net which now is owned by others) to guess how old someone is. At conferences they built a booth that takes a picture.

LEARN: https://docs.microsoft.com/en-us/learn/modules/read-text-computer-vision/

DEMO: Seeing AI app talking camera narrates the world around blind people.

Computer Vision

Computer Vision” analyzes images and video to extract descriptions, tags, objects, and text. API Reference, DOCS, INTRO:

az-ai-produce-objects-372x278

Flow API samples

https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?tabs=visual-studio&pivots=programming-language-csharp READ

Computer Vision demo

  1. Select images and review the information returned by the Azure Computer Vision web service:

    DEMO: https://aidemos.microsoft.com/computer-vision

    1. Click an image to see results of “Analyze and describe images”. Objects are returned with a bounding box to indicate their location within the image.

    2. Click “Try another image” for another selection.
    3. Click “Next step”.
    4. Read text in imagery.
    5. Read handwriting
    6. Recognize celebrities & landmarks - the service has a specialized domain model trained to identify thousands of well-known celebrities from the worlds of sports, entertainment, and business. The “Landmarks” model can identify famous landmarks, such as the Taj Mahal and the Statue of Liberty.

    Additionally, the Computer Vision service can:

    • Detect image types - for example, identifying clip art images or line drawings.
    • Detect image color schemes - specifically, identifying the dominant foreground, background, and overall colors in an image.
    • Generate thumbnails - creating small versions of images.
    • Moderate content - detecting images that contain adult content or depict violent, gory scenes.

    Computer Vision API shows all the features.

HANDS-ON LAB:

  1. Right-click “Launch VM mode” for the “AI-900” lab on a Window VM.
  2. X: Click the Edge browser icon.
  3. X: Click to remove pop-ups.
  4. Go to portal.azure.com
  5. Sign in using an email which you have an Azure subscription.
  6. Type the password. You can’t copy outside the VM and paste into it.
  7. X: Do not save your password.
  8. Open Visual Studio to see

  9. X: On another browser tab, view the repo (faster):

    https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/01%20-%20Image%20Analysis%20with%20Computer%20Vision.ipynb

  10. Follow the instructions in the notebook to create a resource, etc.

    TODO: Incorporate the code and put it in a pipeline that minimizes manual actions.

  11. To take a quiz and get credit, click in the VM “here to complete the Learn module with a Knowledge Check.

Custom Vision

Azure Custom Vision trains custom models referencing custom (your own) images. Custom vision has two project types:

HANDS-ON: LEARN hands-on lab

LAB: Steps:

  1. Create state-of-the-art computer vision models.
  2. Upload and tag training images.
  3. Train classifiers for active learning.
  4. Perform image prediction to identify probable matches to a trained model.

  5. Perform object detection to locate elements within an image and return a bounding box.

  6. Open

    https://www.customvision.ai

    DOCS: az-ai-image-class-623x410

  7. MS LEARN HANDS-ON LAB:

    aka.ms/learn-image-classification which redirects to
    docs.microsoft.com/en-us/learn/modules/classify-images-custom-vision

  8. Instructions are at

  9. Load the code from:

    https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/03%20-%20Object%20Detection.ipynb

https://docs.microsoft.com/en-us/learn/modules/evaluate-requirements-for-custom-computer-vision-api/3-investigate-service-authorization Custom Vision APIs use two subscription keys, each control access to an API:

https://docs.microsoft.com/en-us/learn/modules/evaluate-requirements-for-custom-computer-vision-api/4-examine-the-custom-vision-prediction-api

References: CV API

Face => AI

VIDEO: API DEMO:

Azure “Face” is used to build face detection and facial recognition solutions in five categories:

NOTE: On June 11, 2020, Microsoft announced that it will not sell facial recognition technology to police departments in the United States until strong regulation, grounded in human rights, has been enacted. As such, customers may not use facial recognition features or functionality included in Azure Services, such as Face or Video Indexer, if a customer is, or is allowing use of such services by or for, a police department in the United States.

FaceTutorials

HANDS-ON: LEARN tutorial using these Lab files.

A face location is face coordinates – a rectangular pixel area in the image where a face has been identified.

The Face API can return up to 27 landmarks for each identified face that you can use for analysis. Azure allows a person to can have up to 248 faces. There is a 6 MB limit on the size of each file (jpeg, png, gif, bmp).

Face attributes are predefined properties of a face or a person represented by a face. The Face API can optionally identify and return the following types of attributes for a detected face:

Emotions detected in JSON response is a floating point number:

PROTIP: “Happiness: 9.99983543,” is near certainty at 1.0. 2.80234E-08” indicates 8

https://github.com/Azure-Samples/cognitive-services-FaceAPIEnrollmentSample

DEMO: LAB: https://github.com/microsoft/hackwithazure/tree/master/workshops/web-ai-happy-sad-angry

https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/quickstarts-sdk/identity-client-library?tabs=windows%2Cvisual-studio&pivots=programming-language-csharp

Create a Face API subscription

Subscribe to the Face API:

  1. Sign in to the Azure portal.
  2. Go to
    • Create a resource > AI + Machine Learning > Face</a>

  3. Enter a unique name for your Face API subscription name in variable MY_FACE_ACCT

    Paste in setme.sh

    export MY_FACE_ACCT=faceme
  4. Choose “westus”, the Location nearest to you.
  5. Select F0 the free or lowest-cost Pricing tier.
  6. Check “By checking this box, I certify that use of this service or any service that is being created by this Subscription Id, is not by or for a police department in the United States.”

  7. Click “Create” to subscribe to the Face API.

  8. When provisioned, “Go to resource”.

  9. In “Keys and Endpoint”, copy Key1 and paste in setme.sh

    export MY_FACE_KEY1=subscription_key

The endpoint used to make REST calls is “$MY_FACE_ACCT.cognitiveservices.azure.com/”

https://github.com/wilsonmar/azure-quickly/blob/main/az-face-init.sh

https://docs.microsoft.com/en-us/learn/modules/identify-faces-with-computer-vision/8-test-face-detection?pivots=csharp

References:

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/Overview What is the Azure Face service?

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/

Face API

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/quickstarts/client-libraries?tabs=visual-studio&pivots=programming-language-csharp

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/04%20-%20Face%20Analysis.ipynb

CustomVision

Form Recognizer = Document Intelligence

Video Indexer

Media Services & Storage Account:

  1. In Portal, Media Services blade.

  2. Specify Resource Group, account name, storage account, System-managed identity.

    az-media-service-svcs.png

  3. In a browser, go to the Video Indexer Portal URL:

    https://www.videoindexer.ai

    NOTE: Video Indexer is under Media Services rather than Cognitive Services.

  4. Click the provider to login: AAD (Entra) account, Personal Microsoft account, Google.

    PROTIP: Avoid using Google due to the permissions you’re asked to give:

  5. Say Yes to Video Indexer permission to: Access your email addresses & View your profile info and contact list, including your name, gender, display picture, contacts, and friends.

    NOTE: You’ll get an email with subject “Your subscription to the Video Indexer API”.

    On your mobile phone you’ll get a “Connected to new app” notice for Microsoft Authenticator.

  6. Click “Account settings”. PRICING: up to 10 hours (600 minutes) of free indexing to website users and up to 40 hours (2,400 minutes) of free indexing to API users. Media reserved units are pre-paid. See FAQ

  7. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in

    https://bomonike.github.io/azure-quickly

  8. Switch back.
  9. In Account settings, click “Copy” to get the Account ID GUID in your Clipboard.

  10. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in

    https://bomonike.github.io/azure-quickly

  11. Highlight the sample value and paste (Command+V).

  12. Switch back.

  13. Go to the “Azure Video Analyzer for Media Developer Portal”:DOCS

    api-portal.videoindexer.ai

  14. Click “Sign In”. Click “Profile”

    NOTE: The UI has changed since publication of Microsoft’s tutorial, which says “Go to the Products tab, then select Authorization.”

  15. Click “Show” on the “Primary key” line. Double-click on the subscription key to copy to Clipboard (Command+C).

  16. Switch to the file which defines Azure environment variable VIDEO_INDEXER_API_KEY (in setmem.sh) as described in

    https://bomonike.github.io/azure-quickly

  17. Highlight the sample value and paste (Command+V).

  18. Switch back.

    Upload video using Portal GUI

    DOCS

    Upload video using program

    https://api-portal.videoindexer.ai/api-details#api=Operations&operation=Get-Account-Access-Token

  19. DOCS: Select the Azure Video Indexer option for uploading videos: upload from URL (there is also send file as byte array by an API call, which has limits of 2 GB in size and a 30-minute timeout.

    Program

    az-video-upload.py in https://bomonike.github.io/azure-quickly

  20. Make an additional call to retrieve insights.

  21. Reference existing asset ID

    Search Media files

  22. In “Media files” at https://www.videoindexer.ai/media/library

  23. Click “Samples”, and click on a video file to Play to see the media’s people, topics (keywords).

    NOTE: Search results include exact start times where an insight exists, possibly multiple matches for the same video if multiple segments are matched.

  24. Click a tag to see where it was mentioned in the timeline.

    Alternately, use the API to search: ???

    Model customizations

    Each video consists of scenes grouping shots, which each contain keyframes.

    • A scene represents a single event within the video. It groups consecutive shots that are related. It will have a start time, end time, and thumbnail (first keyframe in the scene).

    • A shot represents a continuous segment of the video. Transitions within the video are detected which determine how it is split into shots. Shots have a start time, end time, and list of keyframes.

    • Keyframes are frames that represent the shot. Each one is for a specific point in time. There can be gaps in time between keyframes but together they are representative of the shot. Each keyframe can be downloaded as a high-resolution image.

  25. Click “Model customizations”

  26. Set as thumbnail another keyframe.

https://github.com/Azure-Samples/media-services-video-indexer

https://dev.to/adbertram/getting-started-with-azure-video-indexer-and-powershell-3i32

Azure Form Recognizer

“Form Recognizer” extracts information from images obtained from scanned forms and invoices.

Resources:

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/06%20-%20Receipts%20with%20Form%20Recognizer.ipynb

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/tables-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-java/formrecognizer-java-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-net/azure-form-recognizer-client-sdk-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/formrecognizer-samples/

OCR

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/05%20-%20Optical%20Character%20Recognition.ipynb

Ink Recognizer

VIDEO:

Ink converts handwriting to plain text, in 63+ core languages.

It was deprecated on 31 January 2021.

https://docs.microsoft.com/en-us/azure/cognitive-services/ink-recognizer/quickstarts/csharp Quickstart: Recognize digital ink with the Ink Recognizer REST API and C#

QUESTION: Does it integrate with a tablet?

Create Cognitive Services

Azure Machine Learning 2.0 CLI (preview) examples

https://github.com/Azure-Samples/Cognitive-Services-Vision-Solution-Templates

BTW https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-services-quickstart-code/cognitive-services-quickstart-code/ https://github.com/Azure-Samples/cognitive-services-sample-data-files

My script does the same as these manual steps:

  1. In Portal.azure.com
  2. G+\ Cognitive Services
  3. Click the Name you created.
  4. Click “Keys and Endpoint” in the left menu.

  5. Click the blue icon to the right of KEY 1 heading to copy it to your invisible Clipboard.

  6. Endpoint: https://tot.cognitiveservices.azure.com/

    TODO: DOCS: Automate above steps to create compute and server startpup script.

    PROTIP: These instructions are not in Microsoft LEARN’s tutorial.

    “Your document is currently not connected to a compute. Switch to a running compute or create a new compute to run a cell.”

  7. Click the Run triangle for “Your document is currently not connected to a compute.”
  8. “Create compute”
    • Virtual machine type: CPU or GPU
    • Virtual machine size: Select from all options (64 of them) QUESTION: What is the basis for “recommended”?
    • The cheapest is “Standard_F2s_v2” with “2 cores, 4GB RAM, 16GB storage” for Compute optimized at “$0.11/hr”. See Microsoft’s description of virtual machine types here.
    • Next
    • Compute name: PROTIP: Use 3-characters only, such as “wow” or “eat”.
    • Enable SSH access: leave unchecked
    • Create
    • Wait (5 minutes) for box to go from “Creating” to “Running”.

Kinds of Cognitive Services CLI

You would save money if you don’t leave servers running, racking up charges.

You can confidently delete Resource Groups and all resources attached if you have automation in CLI scripts that enable you to easily create them later.

Instead of the manual steps defined in this LAB, run my Bash script in CLI, as defined by this DOC:

  1. G+\ Cognitive Services.

  2. Click the ＋Create a resource button, search for Cognitive Services, and create a Cognitive Services resource with the following settings:
  3. Subscription: Your Azure subscription.
  4. Resource group: Select or create a resource group with a unique name.
  5. Region: Choose any available region:
  6. Name: Enter a unique name.
  7. Pricing tier: S0
  8. I confirm I have read and understood the notices: Selected.

TODO: Instead of putting plain text of cog_key in code, reference Azure Vault. Have the code in GitHub.

Azure has a cognitiveservices CLI subcommand.

https://docs.audd.io/?ref=public-apis

AI Language services

Formerly called “NLP” (Natural Language Processing), Intro: Tutorial: https://docs.microsoft.com/en-us/learn/paths/explore-natural-language-processing

NLP enables the creation of software that can:

Within Microoft, NLP consists of these Azure services (described below):

AI Language Tutorials

https://microsoftlearning.github.io/mslearn-ai-language/

Speaker Recognition

Speaker Recognition for authentication.

In contrast, Speaker Diarization groups segments of audio by speaker in a batch operation.

“In cloudinary”

LUIS

Think of “LUIS” as Amazon Alexa’s frienemy.

https://www.luis.ai, provides examples of how to use LUIS (Language Understanding Intelligent Service) thus:

A machine learning-based service to build natural language into apps, bots, and IoT devices. Quickly create enterprise-ready, custom models that continuously improve.

msft-ai-luis-3204x1538.png

Bot Framework Emulator Follow the instructions at https://github.com/Microsoft/BotFramework-Emulator/blob/master/README.md to download and install the latest stable version of the Bot Framework Emulator for your operating system.

Bot Framework Composer Install from https://docs.microsoft.com/en-us/composer/install-composer.

Utterances are input from the user that your app needs to interpret.

https://github.com/Azure-Samples/cognitive-services-language-understanding

CHALLENGE: Add natural language capabilities to a picture-management bot.

https://www.slideshare.net/goelles/sharepoint-saturday-belgium-2019-unite-your-modern-workplace-with-microsofsts-ai-ecosystem

LUIS Regional websites

Perhaps for less latency, create the LUIS app in the same geographic location where you created the service:

Create LUIS resource

NOTE: LUIS is not a service like “Cognitive Services”, but a Marketplace item:

  1. In the Azure portal, select + Create a resource.
  2. In the “Search services and Marketplace” box, type LUIS, and press Enter.
  3. In search results, select “Language Understanding from Microsoft”.
  4. Select Create.
  5. Leave the Create options set to Both (Authoring and Prediction).
  6. Choose a subscription.
  7. Create a new resource group named “LearnRG”.

  8. Enter a unique name for your LUIS service. This is a public sub-domain to a LUIS Regional websites (above)

    PROTIP: Different localities have different costs.

  9. For Authoring Location, choose the one nearest you.
  10. For Authoring pricing tier, select F0
  11. Set your Prediction location to the same region you chose for Authoring location
  12. For Prediction pricing tier, select F0. If F0 is not available, select S0 or other free/low cost tier.
  13. Select Review + Create.
  14. If the validation succeeds, select Create.
  15. After “Your deployment is complete” appears, click “Go to resource” (within Cognitive Services).
  16. Within the Resource Management section, select the “Keys and Endpoints” section.
  17. Click the blue button to the right of KEY 1’s value. That copies it to your Clipboard.

  18. If you are making use of my framework, open the environment variables file:

    code ../setmem.sh

  19. Locate the variables and replace the sample values:

    export MY_LUIS_AUTHORING_KEY=”abcdef1234567896b87f122281e9187e” export MY_LUIS_ENDPOINT=””

    SECURITY PROTIP: Putting key values in a global variables definition file separate from the code.

1.

authoring_key = env:MY_LUIS_AUTHORING_KEY
   authoring_endpoint = env:MY_LUIS_ENDPOINT



  1. Select a LUIS Regional websites (above)
  2. Login.
  3. Choose an authoring resource (Subscription). If there are several listed, select the top one.

  4. Create a new authoring service (Azure service).

    Create a LUIS app in the LUIS portal

  5. Now at https://www.luis.ai/applications, notice at the upper-right “PictureBotLUIS (westus, F0)” where the Directory name usually appears in the Portal.

    The pages displayed will be different if you have already created a LUIS app or have no apps created at all. Select either Go to apps, or the apps option that is available on your initial LUIS page.

  6. Select “+ New app” for conversation Apps for the “Create new app” pop-up dialog.

    The LUIS user interface is updated on a regular basis and the actual options may change, in terms of the text used. The basic workflow is the same but you may need to adapt to the UI changes for the text on some elements or instructions given here.

    Take note of the other options such as the ability to import JSON or LU file that contains LUIS configuration options.

  7. Give your LUIS app a name, for example, PictureBotLUIS.

  8. For Culture, type “en-us” then select the appropriate choice for you language.

  9. Optionally, give your LUIS app a description so it’s clear what the app’s purpose is.
  10. Optionally, if you already created a Prediction Resource, you can select that in the drop-down for Prediction resource.

  11. Select Done.

  12. Dismiss the guidance dialog that may display.

    az-ai-luis-utterances-912x268

    DEFINITION: Each bot action has an intent invoked by an utterance, which gets processed by all models. The top scoring model LUIS selects as the prediction.

    An utterance that don’t map to existing intents is called the catchall intent “None”.

    Intents with significantly more positive examples (“data imbalance” toward that intent) are more likely to receive positive predictions.

    Create a LUIS app with code

  13. On your local machine (laptop), install Visual Studio Code for your operating system.

  14. If you will be completing your coding with Python, ensure you have a Python environment installed locally. Once you have Python installed, you will need to install the extension for VS Code.

    Alternately, to use C# as your code language, start by installing the latest .NET Core package for your platform. You can choose Windows, Linux, or macOS from the drop-down on this page. Once you have .NET Core installed, you will need to add the C# Extension to VS Code. Select the Extensions option in the left nav pane, or press CTRL+SHIFT+X and enter C# in the search dialog.

  15. Create a folder within your local drive project folder to store the project files. Name the folder “LU_Python”. Open Visual Studio Code and Open the folder you just created. Create a Python file called “create_luis.py”.

    cd
cd projects
mkdir LU_Python
cd LU_Python
touch create_luis.py
code .

  16. Install the LUIS package to gain access to the SDK.

    sudo pip install azure-cognitiveservices-language-luis

  17. Open an editor (Visual Studio Code)

    Create entities in the LUIS portal

  18. Be signed in to your LUIS app and on the Build tab.
  19. In the left column, select Entities, and then select + Create.
  20. Name the entity facet (to represent one way to identify an image).

  21. Select Machine learned for Type. Then select Create.

    Create a prediction key

    https://docs.microsoft.com/en-us/learn/modules/manage-language-understanding-intelligent-service-apps/2-manage-authoring-runtime-keys

  22. Sign in to your LUIS portal.
  23. Select the LUIS app that you want to create the prediction key for.
  24. Select the Manage option in the top toolbar.

  25. Select Azure Resources in the left tool bar.

    Unless you have already created a prediction key, your screen should look similar to this. The key information is obscured on purpose.

    A Starter Key provides 1000 prediction endpoint requests per month for free.

    Examples of utterances are on the Review endpoint utterances page on the Build tab.

    Install and run LUIS containers

    docker pull mcr.microsoft.com/azure-cognitive-services/luis:latest

  26. Specify values for Billing LUIS authoring endpoint URI and ApiKey:

    docker run --rm -it -p 5000:5000 --memory 4g --cpus 2 --mount type=bind,src=c:\input,target=/input
--mount type=bind,src=c:\output\,target=/output
mcr.microsoft.com/azure-cognitive-services/luis
Eula=accept
Billing={ENDPOINT_URI}
ApiKey={API_KEY}

    ”–mount type=bind,src=c:\output\,target=/output” indicates where the LUIS app saves log files to the output directory. The log files contain the phrases entered when users hit the endpoint with queries.

  27. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -G \
-d verbose=false \
-d log=true \
--data-urlencode "query=Can I get a 5x7 of this image?" \
"http://localhost:5000/luis/v3.0/apps/{APP_ID}/slots/production/predict"

  28. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -X GET \
"http://localhost:5000/luis/v2.0/apps/{APP_ID}?q=can%20I%20get%20an%20a%205x7%20of%20this%20image&staging=false&timezoneOffset=0&verbose=false&log=true" \
-H "accept: application/json"

DEMO JSON responses

  1. DEMO: voice control lighting in a virtual home.

    https://aidemos.microsoft.com/luis/demo

  2. Select suggested utterances to see the JSON response:

    • Book me a flight to Cairo
    • Order me 2 pizza
    • Remind me to call my dad tomorrow
    • Where is the nearest club?

    Type instructions, use the microphone button to speak commands.

    LUIS identifies from your utterance your intents and entities.

    Entity types

    • List - fixed, closed set of related words (small, tiny, smallest). Case-sensitve
    • RegEx - (credit card numbers)
    • Prebuilt
    • Pattern.Any
    • Machine Learned

LUIS CLI

Run my az-luis-cli.sh.

DOCS:

https://github.com/cloudacademy/using-the-azure-machine-learning-sdk

  1. Create a Resource referenced when LUIS Authoring is defined. The resource name should be lower case as it is used for the endpoint URL, such as:

    https://luis-resource-name.cognitiveservices.azure.com/

  2. The Bot Framework CLI requires Node.js.

    npm i -g npm
  3. Install Bot Framework Emulator per instructions at:
    https://github.com/Microsoft/BotFramework-Emulator/blob/master/README.md

    Download to downloads folder at:
    https://aka.ms/bf-composer-download-mac

  4. Install Bot Framework Composer per instructions at:
    https://docs.microsoft.com/en-us/composer/install-composer

    On a Mac, download the .dmg
    https://aka.ms/bf-composer-download-mac

  5. Use Node.js to install the latest version of Bot Framework CLI from the command line:

    npm i -g @microsoft/botframework-cli

    https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-bf-cli-deploy-luis?view=azure-bot-service-4.0

    LUIS can generate from models a TypeScript or C# typw (program code).

    Luis.ai

  6. DEMO: https://www.luis.ai

  7. Sign in and create an authoring resource refercening the Resource Group.

    PROTIP: In the list of Cognitive services kinds, resources and subscription keys created for LUIS authoring are separate than ones for prediction runs so that utilization for the two can be tracked separately.

    https://aka.ms/AI900/Lab4 which redirects to
    Create a language model with Language Understanding which trains a (LUIS) language model that can understand spoken or text-based commands. He’s Alexa’s boyfriend, ha ha.

  8. Process Natural Lanaguage using Azure Cognitive Language Services

    https://github.com/MicrosoftLearning/AI-102-LUIS contains image files for reference by https://github.com/MicrosoftLearning/AI-102-Code-Repos https://github.com/MicrosoftLearning/AI-102-Process-Speech

    PROTIP: LUIS does not perform text summarization. That’s done by another service in the pipeline.

Terminology:

References:

Text Analytics API Programming

Techniques DOCS:

NOTE: The previous version references interactive Python Notebooks such as MS LEARN HANDS-ON LAB referencing “07 - Text Analytics.ipynb”.

  1. Look at the DEMO GUI at:

    aidemos.microsoft.com/text-analytics

  2. Click “Next Step” through the various processing on a sentence:

    • Sentiment analysis (how positive or negative a document is)

    • Key phrase extraction

    • Entity Linking to show link in Wikipedia.

    • Bing Entity Search

    • Text to Speech services:
    • Language Detection (is it English, German, etc.)
    • Translator Text

API:

Text Analytics API Reference

Some Text Analytics API services are synchronous and asynchronous

Cloud Academy lab “Using Text Analytics in the Azure Cognitive Services API”

docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/textanalytics-samples

Azure Text Analytics client library samples for JavaScript has step-by-step instructions:

  1. Logging into the Microsoft Azure Portal

    <a target=”_blank” href=”https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.CognitiveServices%2Faccounts”“>Cognitive Services</a>

  2. Select the service already defined for you.

    Retrieving Azure Cognitive Services API Credentials

  3. At the left menu click “Keys and Endpoint”. The Endpoint URL contains:

    https://southcentralus.api.cognitive.microsoft.com/

    The Endpoint is the location you’ll be able to make requests to in order to interact with the Cognitive Services API. The Key1 value is the key that will allow you to authenticate with the API. Without the Key1 value, you will receive unauthenticated errors.

  4. In the Azure Portal, type Function App into the search bar and click Function App:

alt

  1. Click the only option:

alt

You’ll be brought to the Function App blade. While a complete summary of function apps is outside the scope of this lab, you should know the function apps allow you to create custom functions using a variety of programming languages, and to trigger them using any number of events.

In the case of this lab step, you’ll set up a function app to interact with the Cognitive Services API using Node.js. You’ll then configure the function to be triggered by visiting its URL in a browser tab.

  1. On the left side of the blade, click on Functions and then + Add.

  2. Click the HTTP Trigger option:

alt

Important: If you don’t see the HTTP trigger option, click More Templates and then Finish and view templates to see the correct option.

  1. Name the function language-detection and click Create:

alt

  1. Click Code + Test in the left sidebar then look at the upper part of the console and switch to the function.json file:

  2. Replace the contents of the file with the following snippet and click Save:

    {
  "bindings": [
 {
   "authLevel": "function",
   "type": "httpTrigger",
   "direction": "in",
   "name": "req",
   "methods": [
     "get",
     "post"
   ]
 },
 {
   "type": "http",
   "direction": "out",
   "name": "$return"
 }
  ],
  "disabled": false
}

The function.json file manages the behavior of your function app.

  1. Switch to the index.js file:

  2. Replace the contents of the file with the following code:

'use strict';
 
let https = require('https');
const subscription_key = "INSERT_YOUR_KEY_HERE";
const endpoint = "INSERT_YOUR_ENDPOINT_HERE";
const path = '/text/analytics/v2.1/languages';
 
module.exports = async function (context, req) {
  let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},
    ]
  };
 
  let body = JSON.stringify(documents);
 
  let request_params = {
    method: 'POST',
    hostname: (new URL(endpoint)).hostname,
    path: path,
    headers: {
      'Ocp-Apim-Subscription-Key': subscription_key,
    }
  };
 
  let response = await makeRequest(request_params, body);
 
  context.res = {
    body: response
  }
 
};
function makeRequest(options, data) {
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      res.setEncoding('utf8');
      let responseBody = '';

      res.on('data', (chunk) => {
        responseBody += chunk;
      });

      res.on('end', () => {
        resolve(JSON.parse(responseBody));
      });
    });

    req.on('error', (err) => {
      reject(err);
    });

    req.write(data)
    req.end();
  });
}
  1. Replace the contents of subscription_key and endpoint with the key and endpoint you retrieved in an earlier step, so that the variables look similar to this:

Copy code 1 2 const subscription_key = “4779caba69344c3d97bca9863d726af6”; const endpoint = “https://southcentralus.api.cognitive.microsoft.com/”;

  1. Click Save.

  2. While a deep understanding of code is outside the scope of this lab, you should take note of a couple of things:

The endpoint and subscription key that you set will allow the function app to communicate with your Cognitive Services API. The endpoint tells the app where to find the API, and the subscription key allows the app to authenticate. The path variable declares the exact path the app should request on the Cognitive Services API. This path system allows the API to offer many different services in one endpoint: Copy code 1 const path = ‘/text/analytics/v2.1/languages’;

The documents variable declares a list of documents that will be passed to the language detection API. The API will return the languages that these documents are most likely written in Currently there are three documents, and the goal will be to get the language used in each one:

let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},
    ]
  };

The rest of the code simply manages the formatting of the data, requesting the language detection service from the API, and returning the result. Next, you’ll visit the function app’s URL to see it work.

  1. Look at the upper command bar and click Test/Run.

  2. Select GET as Http Method before clicking Run:

  3. View the output in the Output tab:

alt

Notice that what’s returned is a JSON object with a “documents” object containing three results, one for each language you submitted to it. Each result has a “name” value with the predicted language and a “score” value with the likelihood that the decision is accurate. A score of 1 means that the Cognitive Services API was completely confident in its language detection. Here’s what the formatted JSON object looks like, for reference:

{
  "documents": [
    {
      "id": "1",
      "detectedLanguages": [
        {
          "name": "English",
          "iso6391Name": "en",
          "score": 1
        }
      ]
    },
    {
      "id": "2",
      "detectedLanguages": [
        {
          "name": "French",
          "iso6391Name": "fr",
          "score": 1
        }
      ]
    },
    {
      "id": "3",
      "detectedLanguages": [
        {
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1
        }
      ]
    }
  ],
  "errors": []
}
  1. Using the Azure Text Analytics API for Language Detection
  2. Using the Azure Text Analytics API for Sentiment Analysis

Sentiment Analysis

API Reference

The output is a number from 0 to 1, with 1 being the most positive language and zero being the most negative opinion expressed.

Key phrase extraction

CHALLENGE: API Reference

Named Entity Recognition (NER) identifies entities in the text and group them into different entity categories, such as organization name, location, event, etc.

"documents": [
    {
      "id": "1",
      "keyPhrases": [
        "world",
        "input text"
      ]
    },

Detect Language

CHALLENGE: API Reference

vs. Content Moderator - Moderate Detect Language to Auto Correct, PII, listid, classify, language. Classification of Profanity returns JSON with several categories:

https://learn.microsoft.com/en-us/training/modules/analyze-text-ai-language/3-detect-language

v2.1 API Testing Console

Input text is in a formatted JSON document files of up to 5,120 characters. Each file can contain up to 1,000 id’s, each associated with a text string.

As with GraphQL, the API returns the detected language and a numeric score between 0 and 1. Scores close to 1 indicate 100% certainty that the identified language is true. A total of 120 languages are supported.

   "id": "3",
      "detectedLanguages": [
        {
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1
        }

Only one language code is returned for each document submitted. Mixed language content within the same document returns the language with the largest representation in the content, but with a lower positive rating, reflecting the marginal strength of that assessment.

Named Entity Recognition

CHALLENGE: API Reference

Sample Response:

{
  "documents": [{
    "id": "1",
    "entities": [{
      "name": "Seattle",
      "matches": [{
        "wikipediaScore": 0.15046201222847677,
        "entityTypeScore": 0.80624294281005859,
        "text": "Seattle",
        "offset": 26,
        "length": 7
      }],
      "wikipediaLanguage": "en",
      "wikipediaId": "Seattle",
      "wikipediaUrl": "https://en.wikipedia.org/wiki/Seattle",
      "bingId": "5fbba6b8-85e1-4d41-9444-d9055436e473",
      "type": "Location"
    }, {
      "name": "last week",
      "matches": [{
        "entityTypeScore": 0.8,
        "text": "last week",
        "offset": 34,
        "length": 9
      }],
      "type": "DateTime",
      "subType": "DateRange"
    }]
  }],
  "errors": []
}

Linked Entities from a well known knowledge base

API Reference

Speech Services

Microsoft has incorporated Immersive Reader throughout their products. Here is a good deep dive about what features are currently available on each product and device type:

ms-ai-immersive-240218-1079x479.png

Speech-to-text (STT) has two different REST APIs:

Microsoft Cognitive Services Speech SDK Samples

LAB:

https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/

https://docs.microsoft.com/en-us/learn/modules/transcribe-speech-input-text/?WT.mc_id=cloudskillschallenge_efc530c5-7105-4c12-8eb3-bc20ae3bee78

Speech-to-text

Transcriptions can be done in real-time or in batch mode.

Batch mode is when audio recordings are stored on a file share, and a shared access signature (SAS) URI is used by a program to asynchronously receive transcription results.

  1. Take the introductory tutorial:

    Introduction to Machine Learning with Hands-On Labs

    https://azure.microsoft.com/en-us/documentation/articles/machine-learning-studio-overview-diagram

  2. Create a model.

  3. Prepare Data:

    As per this video using

    • Clean Missing Data - Clip Outliers
    • Edit Metadata
    • Feature Selection
    • Filter
    • Learning with Counts
    • Normalize Data
    • Partition and Sample
    • Principal Component Analysis
    • Quantize Data
    • SQLite Transformation
    • Synthetic Minority Oversampling Technique

  4. Train the model

    • Cross Validation
    • Retraining
    • Parameter Sweep

  5. Score and test the model.

  6. Make predictions with Elastic APIs

    • Request-Response Service (RRS) Predictive Experiment - Batch Execution Service (BES)
    • Retraining API

https://github.com/timothywarner/ai100/tree/master/Speech-to-Text

Speech Translation (Speech-to-text aka STT)

Get the monthly subscription mobile app on iPhone, Android, or Amazon. It has a Phrasebook of common phrases.

DEMO: Speech Translation recognizes and synthesizes speech, and translates spoken languages. REMEMBER: The sequence of services involves two APIs:

az-speech-translate-1756x580

Speech-to-Text API -> Speech Correction -> Machine Translation -> Text-to-Speech API

Azure Cognitive TTS Samples

“Speech Recognition” and Text Analysis are not involved in this use case.

Telephone voice menus use “Speech Synthesis”, defined by the Speech Synthesis Markup Language (SSML).

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/08%20-%20Speech.ipynb

https://github.com/timothywarner/ai100/tree/master/Speech-to-Text

https://github.com/MicrosoftLearning/AI-SpeechToText

The speech-to-text service includes multiple pre-defined voices with support for multiple languages and regional pronunciations, with language detection. In addition to standard voices, neural voices leverage neural networks to overcome common limitations in speech synthesis with regard to intonation, resulting in a more natural sounding voice.

PROTIP: Neural voices are created from samples that use a 24 khz sample rate.

Speech recognition can use a acoustic model of phonemes (sounds) or a language model that matches phonemes with words.

Custom voices can be created with the text-to-speech API.

PROTIP: Since you have to use your own subscription to follow this tutorial from Microsoft, skip clicking “Launch VM mode” and follow the Python notebook on Speech on the regular Portal.

SAMPLE: https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-speech-tts/azure-cognitive-tts-samples/

  1. PROTIP: In a CLI window, run my Bash shell script to Create a Cognitive Services resource and get its two keys:

    cd ~/clouddrive/azure-quickly
git pull
./az-cog-cli.sh

    To synthesize speech, the system typically tokenizes the text to break it down into individual words, and assigns phonetic sounds to each word. It then breaks the phonetic transcription into prosodic units (such as phrases, clauses, or sentences) to create phonemes that will be converted to audio format. These phonemes are then synthesized as audio by applying a voice, which will determine parameters such as pitch and timbre; and generating an audio wave form that can be output to a speaker or written to a file.

    Text-to-speech

To specify that the speech input to be transcribed to text is in an audio file, use AudioConfig.

Change the voice used in speech synthesis by setting the SpeechSynthesisVoiceName property of the SpeechConfig object to the desired voice name.

Speech Translation

Text (text-to-text aka TTT)

Microsoft’s Translator service can translate text between more than 90 languages and dialects (including Klingon in Star Trek), specified using ISO 639-1 two-letter language codes and 3166-1 cultural codes such as “en-US” for US English, “en-GB” for British English, “fr-CA” for Canadian French, etc.

Hands-on tool without a compute instance:

  1. translator.microsoft.com

  2. Click on “Start conversation”, log in and enter your name and language.

  3. Share the conversation code with other participants, who can join using the Micreosoft Translator app or website.

  4. Speak or type in your language to communicate with other participants in the conversation. Other participants will see your messages in their own language.

https://docs.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/overview What is Custom Translator?

Foe Parallel Data, equivalent documents in different languages:

https://docs.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/how-to-upload-document

09 - Translation.ipynb

LEARN: Translate text with Azure AI Translator service

BLAH: You are asked to use your own Subscription anyway, so instead of the Exercise - Translate text and speech, use portal.azure.com directly.

A Python program can run from your laptop or mobile phone making API calls to the Translator endpoint at:

https://api.cognitive.microsofttranslator.com/translate?api-version=3.0

DOCS, API DOCS

VIDEO; Raza Salehi’s 1 hr video course “Build a Translator system”.

VIDEO intro with sample code at https://github.com/microsoft/text-analytics-walkthrough

Python notebook to Translate

For response “script” : “Latn”, text was transliterated in English.

A custom translator is needed to train a model to recognize and translate domain-specific words and phrases in specific industries such as aerospace, automotive, chemistry, mechanical, etc.

portal.customtranslator.azure.ai

Training is done by have pairs of documents (English and French, etc.).

10,000 aligned parallel sentences are neede to train a translator.

In addition to Microsoft Office formats, files with extension .ALIGN for parellel languages are perfectly aligned. Translation Memory systems can export parallel documents in XLF, XLIFF, TMX, suffix. Microsoft’s LocStudio files have .LCL suffix.

Translation runs can each take several hours. So batch processing is supported.

If you don’t have admin

Need admin approval
Mt Studio Web Prod
Mt Studio Web Prod needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.

References:

Immersive Reader

Other sample code

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/documenttranslation-samples/

Conversational AI

HISTORY: In 2015, Microsoft unleashed the Tay chat bot, then had to bring it down after hackers submitted enough racial slurs that they fooled the system into thinking that was normal and acceptable.

HISTORY: XiaoIce, a chatbot Microsoft launched in China, “has more than 200 million users, has engaged in 30 billion conversations, and has an average conversation length of 23 turns, which averages out to about half an hour, achieving human parity at translation from Chinese to English. Japan-based Rinna and the US-based Zo)

Explore-conversational-ai

A Bot Framework enables the creation of Virtual Assistant

A LUIS app creates these types of entities:

https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/concepts/plan?tabs=v1

Bots are extended by Skills

QnA Maker

The cognitive service name “QnA Maker” (Question and Answer Maker)</a> is a cloud-based API service that lets you create a conversational question-and-answer layer over your existing data. The service enables the building of knowledge bases of questions and answers that form the basis of a dialog between a human and an AI agent.

Microsoft created the QnA Maker portal to make it easier than writing code to create and manage knowledge bases using the QnA Maker REST API or SDK.

The knowledge base gets smarter as it continually learns from user behavior.

The knowledge base can be built by extracting questions and answers from your semi-structured content, including FAQs, manuals, and documents.

QnA Maker limits control the size of Knowledge base.

### Create QnA Service

https://github.com/Microsoft/BotBuilder-CognitiveServices/tree/master/CSharp/Samples/QnAMaker is only for C#.

  1. View the DOCS:

    View the v2 (previous release)

  2. The Jupyter notebook:

    https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/11%20-%20QnA%20Bot.ipynb

  3. On an appropriate browser profile (personal or work email) associated with the subscription you wnat to use.

  4. Go to the QnA Maker portal at:

    https://qnamaker.ai

  5. Sign in.

    https://docs.microsoft.com/en-us/learn/paths/explore-conversational-ai/

  6. “Create a knowledge base” from the top menu.
  7. STEP 1: “Create a QnA service”, which opens another tab

  8. Basics: During testing, do NOT click the checkbox for “Managed”. In prod, telemetry and compute are included automatically with your QnA Maker resource. If you do not select managed, you will be prompted to create an App Insights and App Service resources for the required telemetry and compute that you will have to manage for your QnA Maker resource. Read more <a target=”_blank” href=”“https://aka.ms/qnamaker-createoptions-description”>here</a>.

  9. Subscription: Your Azure subscription
  10. Resource group: Select an existing resource group or create a new one

  11. Resource group location: “(US) West US”

  12. Name: A unique name for your QnA resource (App name)
  13. Pricing tier: both are “3 transactions per second, 100 transactions per minute”:
    • Free F0 (3 managed documents per month, 5…)
    • Standard S0 ($10 per month for unlimited documents …)

    App Service details - for runtime :

  14. Azure Search location: Any available location
  15. Azure Search pricing tier: F
    • Free F (50 MB in 3 indexes)
    • Basic B (2 GB in 15 indexes, 3 scale-out units per service on 1 partition with 3 replicas)
    • Standard S1 (25 GB in 50 indexes) - default
    • Standard S2 (100 GB in 200 indexes)
    • Standard S3 (200 GB in 200 indexes)
    • Standard S3 (1000 Index/Partition)
    • L1 (1 TB in 10 indexes, High Storage, 36 units)
    • L2 (2 TB in 10 indexes, High Storage, 36 units)

    Note: If you have already provisioned a free-tier QnA Maker or Azure Search resources, your quota may not allow you to create another one. In which case, select a tier other than F0 / F.

    App Service details - for runtime :

  16. App name: Same as Name (“.azurewebsites.net” will be appended automatically)

  17. Website location: Same as Azure Search location

    App insights details - for telemetry and chat logs :

  18. App insights: Disable, which will hide the “App insights location”, but appear in Review.

  19. Click “Review + create”. Create. NOTE: when these resources complete deployment:
    • Microsoft.Web/sites/config
    • Microsoft.CognitiveServices/accounts
    • Microsoft.Search/searchServices
    • microsoft.insights/components
    • microsoft.insights/components
    • Microsoft.Web/sites
    • Microsoft.Search/searchServices
    • Microsoft.Web/serverfarms

  20. While you wait for the dots to stop flashing “Deployment in progress”, return to the QnA Maker portal tab. You may have timed out.

  21. When “Your deployment is complete”, click “Go to resources” for “Congratulations! Your keys are ready.”

    Connect QnA service to KB

    STEP 2: Connect your QnA service to your KB.

  22. “Refresh” the list of available QnA service resources.
  23. Microsoft Azure Directory ID: The Azure directory ID for your subscription
  24. Azure subscription name: Your Azure subscription

  25. Azure QnA service: The QnA service resource you created in the previous step

    NOTE: In the Preview there is a checkbox “Enable language setting per knowledge base”.

  26. Language: English

    STEP 3: Name your KB.

  27. Type a name: For example: “Margie’s Travel KB”. Spaces are allowed?

    STEP 4: Populate your KB.

  28. ”+ Add URL”

  29. Copy and paste this example URL:

    https://github.com/MicrosoftDocs/ai-fundamentals/raw/master/data/qna_bot/margies_faq.docx

  30. Add file

  31. chit-chat: Adding “chit-chat” to your knowledge base (by selecting a personality) automatically adds questions and responses to your knowledge base, which enables your bot to answer small-talk questions in a voice (personality) of your choice:
    • None
    • Professional « select this.
    • Friendly
    • Witty
    • Caring
    • Enthusiastic

    QUESTION: What is the range of popularity?

    QUESTION: Extraction? I’m stuck here.

  32. Do NOT check “Enable multi-turn extraction from URLs, .pdf or .docx files.”

  33. Click “Create your KB”. Wait for a minute or so while your Knowledge base is created.

  34. Review the questions and answers that have been imported from the FAQ document and the professional chit-chat pre-defined responses.

    Test the knowledge base

    Make an API call

    https://go.microsoft.com/fwlink/?linkid=2100125

    https://go.microsoft.com/fwlink/?linkid=2100213 Coding</a>

Decision (Making)

Anomaly Detector

Anomaly Detector will be retired on 10/1/2026.

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-net/azure-anomaly-detector-client-sdk-samples/

Among Anomaly Detector API Samples

Anomaly Detector identifies potential problems early on.

https://docs.microsoft.com/en-us/learn/modules/get-started-ai-fundamentals/3-understand-anomaly-detection

  1. Sensors in the car collect telemetry, such as engine revolutions, brake temperature, and so on.
  2. An anomaly detection model is trained to understand expected fluctuations in the telemetry measurements over time.
  3. If a measurement occurs outside of the normal expected range, the model reports an anomaly that can be used to alert the race engineer to call the driver in for a pit stop to fix the issue before it forces retirement from the race.

Resources:

Content Moderator

Content Moderator services detect potentially offensive or unwanted content.

This has been deprecated.

Evaluate, Find Faces, Match, OCR.

https://www.youtube.com/watch?v=gVFiA6ZQNAw

CHALLENGE: DOCS:

https://docs.microsoft.com/azure/cognitive-services/Content-Moderator/overview?WT.mc_id=Portal-Microsoft_Azure_Support#data-privacy-and-security

https://docs.microsoft.com/en-us/azure/cognitive-services/content-moderator/client-libraries?tabs=visual-studio&pivots=programming-language-csharp

Response from the Text Moderation API include:

Content Moderation (Evaluate, Find Faces, Match, OCR)

  1. G+/ to Content Moderator”

  2. “Create Custom Moderator”

    Oops!

    Could not create the marketplace item

    This marketplace item is not available.

    When working:

  3. Select Create.

  4. Enter a unique name for your resource, select a subscription, and select a location close to you.

  5. Select the pricing tier for this resource, and then select F0.

  6. Create a new resource group

  7. Select Create.

Metrics Advisor

Metrics Advisor monitors metrics and diagnoses issues.

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/metricsadvisor-samples/

Personalizer

Personalizer creates rich, personalized experiences for every user.

knowledge-mining Tutorials

Knowledge Mining Solution Accelerator

HANDS-ON: Tutorial references these labfiles

  1. Get on the Bing Resource portal GUI.

  2. Define a Resource Group.

  3. Price Tier: Free

    PROTIP: Autosuggest requires the “S2” (Standard) pricing tier. Spell Check requires either S1 or S2.

  4. Click “Automation Options” link for a JSON Template.
  5. Click “Create” at the top.

  6. Create

    Notice the service name at the upper left is “Microsoft.BingSearch” and has a Global location. It’s Endpoint is: https://api.bing.microsoft.com/

Search for AI enrichment of docs

The Azure Cognitive Search service uses a Cognitive Search resource to support AI-powered search and knowledge mining solutions such as:

https://blog.api.rakuten.net/top-10-best-search-apis/

“Document cracking” during indexing extracts text content from unstructured text or non-text content (such as images, scanned documents, or JPEG files). The indexer accesses an Azure data storage service.

https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

Other services

HealthInsights

Bot Service

provides a platform for creating, publishing, and managing bots. Developers can use the Bot Framework to create a bot and manage it with Azure Bot Service - integrating back-end services like QnA Maker and LUIS, and connecting to channels for web chat, email, Microsoft Teams, and others.

Microsoft Bot Framework supports two approaches to integrate bots with agent engagement platforms such as Customer support service:

  1. DEMO: See a healthcare bot built using the Azure Bot Service:

    <a target=”_blank”” href=” https://www.microsoft.com/research/project/health-bot/”> https://www.microsoft.com/research/project/health-bot</a>

    Select the option to Try a demo of an example end-user experience. Use the web chat interface to interact with the bot.

MS LEARN: Create a Bot with the Bot Framework Composer

  1. Run the Python Jupyter notebook

    https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/11%20-%20QnA%20Bot.ipynb Conversational AI LAB

  2. Sign in using the Microsoft account associated with your Azure subscription.

PROTIP: Use NVM to install Node

https://github.com/Microsoft/botbuilder-tools#install-cli-tools says to install Node.js version 10.14.1 or higher

https://github.com/microsoft/botframework-cli says to install Node.js version 12

Since the current version is now 16, we cannot use the command suggested in the doc:

Bot Framework

  1. bf is the bot framework CLI command

    One-stop-shop CLI to manage your bot’s resources. BF CLI and AZ CLI together cover your end-to-end bot development workflow needs.
 
VERSION
  @microsoft/botframework-cli/4.13.3 darwin-x64 node-v16.1.0
 
USAGE
  $ bf [COMMAND]
 
COMMANDS
  chatdown      Converts chat dialog files in <filename>.chat format into
             transcript files. Writes corresponding <filename>.transcript
             for each .chat file.
  config        Configure various settings within the cli.
  dialog        Dialog related commands for working with .schema and .dialog
             files.
  help          display help for bf
  lg            Parse, collate, expand and translate lg files.
  luis          Manages LUIS assets on service and/or locally.
  orchestrator  Display Orchestrator CLI available commands
  plugins       Install, uninstall and show installed plugins
  qnamaker      QnA Maker

References:

Azure OpenAI

References:

OpenAI is a San Francisco-based artificial intelligence research laboratory. OpenAI was founded by Elon Musk, Sam Altman, Greg Brockman, and Ilya Sutskever in December 2015 (to compete with Google’s DeepMind acquisition).

In 2019 Microsoft invested $1 billion in the company as time on Azure cloud and to develop a large-scale AI supercomputer built exclusively for OpenAI’s research in Azure. Azure powers all of OpenAI’s workloads.

In 2023, OpenAI exclusively licensed (closed-source) GPT-3 to Microsoft for their products and services.

In 2022, OpenAI made available their GPT-3.5 foundation model for free trial, offering several categories of capabilities

Source: https://openai.com/blog/openai-microsoft/

OpenAI’s avowed mission is to create Artificial General Intelligence (AGI) (to rival human ability).

OpenAI is a “separate” service from Azure Cognitive Services because now “traditional” Azure Cognitive Services focuses on making predictions based textual and discrete data where OpenAI added to ML “attention” algorithms working on binary data (voice, images, and video) to enable it to perform “Generative AI” which produces new content based on what is described in the input.

  1. Assuming that you have a Azure account and subscription at the Azure portal https://portal.azure.com

  2. Consider pricing at https://azure.microsoft.com/pricing/details/cognitive-services/openai-service

    OpenAI put a paywall behind its more advanced GPT-4 models,

    Billing is based on 1,000 “tokens” increments, with the first 100,000 tokens per month free. Beyond that:

    “Standard” use of the older/more limited gpt-3.5-turbo model is then $0.002 per 1,000 tokens.

    Charges for the GPT-4 model has two dimensions: process stage and the size of the foundational model used :

    • The   “8K context” has   13 billion parameters.

    • The “32K context” has 175 billion parameters.

    • Prompt
    • Completions

    Images generated using DALL-E are $2 per 100 images.

    Available for free for the first 100,000 tokens per month, then $0.004 per 1,000 tokens.

  3. Apply for access to Asure OpenAI for your Region and Currency:

    https://aka.ms/oaiapply

OpenAI Tutorials

OpenAI service features and use cases

Data Science VM

https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/overview#whats-included-in-the-data-science-vm

Resources

Microsoft AI ML Community in Signapore

If you have an OReilly.com account:

On Udemy:

Notes to be inserted

Steps for data transformation:

https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/limits-and-quotas

https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/the-ai-study-guide-azure-machine-learning-edition/ba-p/4063656

https://azure.github.io/Cloud-Native/ https://github.com/azure/cloud-native is a showcase on Azure Cloud Native, the products, events and how to get started or go deep with cloud native technologies, including Serverless on Azure.

Responsible AI

LEARN:

Using Microsoft’s API algorithms and data (such as celebrity faces, landmarks, etc.) means there has be some vetting by Microsoft’s FATE (Fairness, Accountability, Transparency, and Ethics) research group in NYC: az-ai-principled-1126x610 Microsoft’s ethical principles guiding the development and use of artificial intelligence with people:

Resources:

Mehrnoosh Sameki has a https://cs-people.bu.edu/sameki/ResponsibleAI.html course in Responsible AI

“AI Alignment” refers to unintended consequences.

VIDEO: Another OpenAI Scientist QUITS —Says AGI Is a ‘TICKING TIME BOMB’

For Azure AI Foundry Models you pay for:

Lee Stott wrote in Oct 2024 “The Future of AI” says:

To fine tune your own model, follow https://aka.ms/ignite/pre016 which shows you how to fine tune models using Azure AI Foundry 1 click fine tuning or Microsoft Olive Pipeline fine tuning.

but you can also use the tool to fine tune model from hugging face.

GitHub Models are available at https://github.com/marketplace/models and each model card displays the quota/capacity limit

Advanced RAG

Yujian Tang wrote in Oct 2024 “Advanced RAG Applications with Vector Databases” makes use of Python 3.11 on VScode.

More

This is one of a series on AI, Machine Learning, Deep Learning, Robotics, and Analytics:

  1. AI Ecosystem
  2. Machine Learning

  3. Testing AI

  4. Microsoft’s AI
  5. Microsoft’s Azure Machine Learning Algorithms
  6. Microsoft’s Azure Machine Learning tutorial

  7. Microsoft’s Azure Machine Learning certification

  8. Python installation

  9. Juypter notebooks processing Python for humans

  10. Image Processing
  11. Tessaract OCR using OpenCV

  12. Amazon Lex text to speech

  13. Code Generation

  14. Multiple Regression calculation and visualization using Excel and Machine Learning
  15. Tableau Data Visualization