What is Azure Search Cognitive Services?

Azure Cognitive Search (formerly known as Azure Search) is a fully managed service that reduces the complexity of data ingestion and index creation by offering index functionality exposed through a simple REST API or .NET SDK. You can create a search service in minutes and scale it up or down as needed, without worrying about infrastructure or maintenance.

Azure Cognitive Search supports various types of data sources, such as Azure Blob Storage, Azure Cosmos DB, Azure SQL Database, Azure Data Lake Storage, and more. You can use indexers to automate data ingestion and synchronization from these sources, or upload your own JSON documents directly to the service.

Azure Cognitive Search also provides a rich set of features to enhance your search experience, such as:

• Auto-complete, suggestions, synonyms, and spell check to help users find what they are looking for
• Filtering, faceting, sorting, and paging to refine and navigate the search results
• Scoring profiles and ranking models to customize the relevance of the results
• Geo-search and geo-fencing to enable location-based queries
• Security trimming and encryption to protect your data and comply with regulations

Source: Types of decision APIs and Applied AI Services - Azure Architecture Center | Microsoft Learn

How does Azure Cognitive Search use AI?

Azure Cognitive Search leverages the different cognitive services discussed in previous posts such as: speech, vision, and language services to transform raw, unstructured information into searchable content. You can apply cognitive skills to your data before it is indexed by defining your own skillset in the portal or through the API.

Cognitive skills are pre-built or custom functions that extract knowledge from your data, such as:

• OCR (optical character recognition) to extract text from images and PDFs
• Translation to convert text from one language to another
• Key phrase extraction to identify the main topics in a document
• Entity recognition to detect people, places, organizations, dates, etc.
• Sentiment analysis to determine the tone of a text
• Image analysis to extract tags, captions, faces, etc. from images

The output of the cognitive skills is stored as structured fields in your search index, which you can use for filtering, faceting or full-text search. You can also enrich your data with custom skills that you create using Azure Machine Learning or Azure Functions.

How does Azure Cognitive Search support semantic search?

Semantic search is a capability that uses advanced machine learning techniques to understand user intent and contextually rank the most relevant search results. Semantic search goes beyond keyword matching and leverages deep learning models from Microsoft Research and Bing to provide natural language understanding and summarization.

With semantic search, you can:

• Use natural language queries that express your intent rather than specific keywords
• Get summary answers that provide quick snippets of information without clicking on links or documents
• Receive more relevant results that are ranked based on semantic similarity rather than term frequency

Semantic search is available as a preview feature that you can enable on your existing or new indexes. You can also use the semantic ranking model in combination with your own scoring profiles and ranking models.

How can any of these services help you in your business?

Democratization of AI has been a trend in the past few years & is now stronger than ever. Complex AI & ML models are now readily available and allow any business to easily integrate those models in applications, processes, etc. As the technology is now widely available, it is not technology, but your own creativity that limits the value you will be extracting from AI in the next few years. Here is a selection of use cases related to Cognitive Search that could drive value for your organization.

Here are some examples of how you can use the Azure Cognitive Search Service features in your business:

• A healthcare company could use Azure Search Cognitive Services to index and search through medical records, reports, images, and videos. They could also use cognitive skills to extract relevant information from the content, such as patient names, diagnoses, medications, etc.
• A legal company could use Azure Search Cognitive Services to create a document management system that allows them to search and access legal documents, contracts, cases, and precedents. They could also use cognitive skills to analyze the content for sentiment, key phrases, entities, etc.
• A manufacturing company could use Azure Search Cognitive Services to create a knowledge base for their engineers and technicians. They could also use cognitive skills to process technical manuals, schematics, diagrams, and videos for text, image, and speech recognition.
• An education company could use Azure Search Cognitive Services to create a learning platform that offers courses and materials to their students. They could also use cognitive skills to translate the content into different languages, generate captions and summaries, and provide feedback and suggestions.

 So again, your own creativity is your limit!

How can you get started? And what do you need?

To get started with Azure Cognitive Search, you need an Azure subscription and a resource group. You can then create a search service in the portal or through the CLI. You can choose from different pricing tiers based on your needs and budget.

Once you have a search service, you can create an index schema that defines the fields and attributes of your data. You can then populate your index with data from one or more sources using indexers or direct uploads. You can also apply cognitive skills to enrich your data during indexing.

To query your index, you can use the portal’s search explorer tool or the REST API or .NET SDK. You can also integrate your index with your application using code samples and tutorials for various platforms and languages.

You can monitor and manage your search service using metrics, alerts, logs, backups, and more. You can also optimize your search experience using features such as scoring profiles, ranking models, synonyms, etc.

It’s important to carefully plan and design your use case to determine which Azure resources you will need to achieve your desired outcome.

What does it cost?

As it is a cloud service, you pay as you go based on how much you use and which services you use. For the search models, the count is based on documents, queries, and storage.

Here is a non-exhaustive list of the pricing of Azure Search Cognitive Service (note that you get a free tier with 10 000 documents and 50 MB storage):

• For Basic tier, you pay ~ € 75 per month for one search unit that can handle up to 1 million documents and 2 GB storage. You can scale up to 3 search units for higher performance and availability.
• For Standard tier, you pay ~ € 250 per month for one search unit that can handle up to 15 million documents and 25 GB storage. You can scale up to 36 search units for higher performance and availability.
  • You also get additional features such as cognitive skills, synonyms, geo-search, etc.
• You can continue scaling up for more capacity on both the number of documents and/or storage as well as increase towards a high-density tier. For more information, contact us.
• Finally, there are a number of additional features you can activate and which are billed separately:
   
  • Custom Entity Lookup

    • This feature looks for text from a custom, user-defined list of words and phrases and labels all documents containing matching entities. You would use this if you want to define and detect specific entities in your data.

    • Pricing: ~ € 1 per 1000 text records for the first million text records. The price is decreasing for more text records (eg € 0,7 per 1000 text records for one million - 3 million text records).

  • Document Cracking: Image Extraction

    • This feature extracts content from a file where there are images in your documents. Text extraction is free. You would use it if you have documents that contain images.

    • Pricing: ~ € 1 per 1000  trasactions for the first million images. The price is decreasing for more images (eg € 0,75 per 1000 images for one million - 3 million images).

  • Semantic Search

    • This feature uses AI models to improve the relevance of the search results by finding content that is semantically similar to query terms. The service is only available for accounts at minimum on Standard tiers. You would use this feature if you want to improve the quality of search results and optimize the user experience.

    • Pricing: you pay € 467 per month for up to 250k queries. For each additional query on top, you pay € 0,002. Note that you get up to 1000 semantic queries per month for free. 

A document is a unit of searchable data in your index. A query is a request to your search service to retrieve or update documents. Storage is the amount of disk space used by your indexes and data sources. A search unit or a partition is a combination of CPU and memory resources allocated to your search service.

Conclusion

Azure Search Cognitive Services is a cloud search service that gives you the power of AI to enrich and explore your data at scale. It simplifies the data ingestion and index creation process, and provides a rich query language and API to retrieve and rank the most relevant results. It also supports semantic search, which uses deep learning models to understand user intent and context. Do not hesitate to reach out and let's discuss how it would practically work in your environment.