Today we are going to see about the concept of chunking the PDF data passing it to the Embedding model.
In my previous blog post, we have seen the chunking methods.
1) Traditional chunking [Fixed size]
2) Overlap[split few words earlier]
3) Semantic
4) Embebbed
5) sliding chunking
6) Token based
Today we are going to see the sliding chunking.
Sliding Chunking
If you have a content of [0-100] chars then data will be splitted like [0-30,20-50,40,60 ..., ] this gap we have provided like 20 is the stepsize.
Problem:
if both the lines are not related then overlapping concept will be a failure.
while splitting, the values of the embedded points are stored in nearby. so we will not get any new context while responding.
when to consider:
If the content is not related to each other and splitted data(no relationship between sentences).
To create relationship between unstructured data then we can go with this approach.
Token based
input you are converting like a tokens.
they will create a hashmap, they will map all the values in a text to assigned values.
IF the model is response based on the token then this method will be used. any rating limit is set then we need to send based on token for embedding.
This is used very less in real time so concept is more than enough.
TOON: Token Object Oriented Notation
This is an alternative to JSON for AI field.
If you have a JSON structure, then you can cross check with TOON structure. We can reduce the size and structure will be changed.
but still people are working on the TOON structure revamping still the using the structure is bit tricky one.
----
READ PDF
To read PDF we need to use Multiple packages to read data. because
PDF can contains images, emoji, scanned copies, different file content structure, still more options will be there in the PDF copy.
LaTex coding - which is like a formula in the PDF document.
Below packages will helpful for reading the PDF.
PYPDFloader, Camelot (table and formatting), pytesseract, PDFplumber
LLMLingua - check this out as well for additional learning purpose.
So far we are seeing the text based data form a PDF copy splited into chunks and how to store in Vector DB.
Above image is from google source
Comments
Post a Comment