Cleaned up Literature folder
This commit is contained in:
parent
73a6380034
commit
fe5eda4e05
586 changed files with 53911 additions and 2475 deletions
|
|
@ -12,4 +12,4 @@ From a [LinkedIn post](https://www.linkedin.com/feed/update/urn:li:activity:7272
|
|||
|
||||
4. Identify levers: What actions (preemptive or reactive) are you able to take or plan? Probably only a handful. The good news is that your levers almost certainly mitigate >90% of the negative reputation outcomes.
|
||||
|
||||
|
||||
|
||||
7
Corpus/Various/Behavioural change theories.md
Normal file
7
Corpus/Various/Behavioural change theories.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Behavioural change theories
|
||||
|
||||
[This Wikipedia page](https://en.wikipedia.org/wiki/Behavioural_change_theories) describes several Behavioural change theories, including:
|
||||
- [Fogg Behavior Model](Fogg%20Behavior%20Model.md)
|
||||
- [Theory of planned behavior](Theory%20of%20planned%20behavior.md)
|
||||
|
||||
|
||||
|
|
@ -4,4 +4,4 @@ How do the controls fit in with DevSecOps?
|
|||
Related:
|
||||
[ISO 27001 A.14.2.8 System security testing](../../Standards/ISO27x/legacy/ISO%2027001%202013/ISO%2027001%20A.14.2.8%20System%20security%20testing.md)
|
||||
[ISO 27001 A.14.2.9 System acceptance testing](../../Standards/ISO27x/legacy/ISO%2027001%202013/ISO%2027001%20A.14.2.9%20System%20acceptance%20testing.md)
|
||||
[Red, Blue, and Purple Teams](../../Literature%20notes/Red,%20Blue,%20and%20Purple%20Teams.md)
|
||||
[Red, Blue, and Purple Teams](../../Information%20Security/Red,%20Blue,%20and%20Purple%20Teams.md)
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
# Secure Development Lifecycle (SDLC)
|
||||
|
||||
|
||||
https://hackernoon.com/what-is-secure-sdlc
|
||||
|
||||
related to [a-8.25-Secure-development-life-cycle](../../Standards/ISO27x/OST/27002/EN/a-8.25-Secure-development-life-cycle.md)
|
||||
|
||||
|
||||
10
Corpus/Various/Business processes/Security in Development.md
Normal file
10
Corpus/Various/Business processes/Security in Development.md
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[OWASP Top 10 CI-CD Security Risks](../../Standards/other/OWASP%20Top%2010%20CI-CD%20Security%20Risks.md)
|
||||
|
||||
[a-8.25-Secure-development-life-cycle](../../Standards/ISO27x/OST/27002/EN/a-8.25-Secure-development-life-cycle.md)
|
||||
[a-8.26-Application-security-requirements](../../Standards/ISO27x/OST/27002/EN/a-8.26-Application-security-requirements.md)
|
||||
[a-8.27-Secure-system-architecture-and-engineering-principles](../../Standards/ISO27x/OST/27002/EN/a-8.27-Secure-system-architecture-and-engineering-principles.md)
|
||||
[a-8.28-Secure-coding](../../Standards/ISO27x/OST/27002/EN/a-8.28-Secure-coding.md)
|
||||
|
||||
|
||||
NSA has released guidance asking companies to switch to memory-safe languages [PDF](https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI_SOFTWARE_MEMORY_SAFETY.PDF)
|
||||
[Memory safety Wikipedia](https://en.wikipedia.org/wiki/Memory_safety?wprov=sfti1)
|
||||
Binary file not shown.
BIN
Corpus/Various/Business processes/secure_data_lifecycle.jpeg
Normal file
BIN
Corpus/Various/Business processes/secure_data_lifecycle.jpeg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 146 KiB |
11
Corpus/Various/Corporate social responsibility.md
Normal file
11
Corpus/Various/Corporate social responsibility.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
Corporate social responsibility (CSR) is traditionally broken into four categories: environmental, philanthropic, ethical, and economic responsibility.
|
||||
|
||||
**Environmental responsibility** refers to the belief that organizations should behave in as environmentally friendly a way as possible.
|
||||
|
||||
**Ethical responsibility** is concerned with ensuring an organization is operating in a fair and ethical manner.
|
||||
|
||||
**Philanthropic responsibility** refers to a business’s aim to actively make the world and society a better place.
|
||||
|
||||
**Economic responsibility** is the practice of a firm backing all of its financial decisions in its commitment to do good in the areas listed above.
|
||||
|
||||
Source: [Harvard Business School](https://online.hbs.edu/blog/post/types-of-corporate-social-responsibility), retrieved December 13, 2021
|
||||
|
|
@ -74,8 +74,8 @@ Prompt: `elaborate on designing the slot configuration schema`
|
|||
Here's a detailed approach to designing a flexible slot configuration schema that can handle various conversation types and business requirements:
|
||||
|
||||
## Slot Configuration Database Schema
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
This ERD shows the complete database schema for the conversational agent's slot configuration system. Here's a breakdown of the key relationships and design decisions:
|
||||
|
||||
|
|
@ -98,7 +98,7 @@ This ERD shows the complete database schema for the conversational agent's slot
|
|||
**slot_collection_attempts** provides detailed audit trail of user interactions for analytics and debugging
|
||||
|
||||
Let me show you a Python implementation that demonstrates how to work with this schema:
|
||||
|
||||
|
||||
|
||||
### Key Design Features
|
||||
|
||||
|
|
|
|||
29
Corpus/Various/DPOaaS offer Glownexus.md
Normal file
29
Corpus/Various/DPOaaS offer Glownexus.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
We as Glownexus provide a tailored DPOaaS offering which fully covers the requirements of Article [37](https://gdpr-info.eu/art-37-gdpr/), [38](https://gdpr-info.eu/art-38-gdpr/) and [39](https://gdpr-info.eu/art-39-gdpr/) of the EU GDPR which also covers the UK (DPA 2018).
|
||||
|
||||
The DPOaaS will guide Omeros through all the important tasks Omeros has to do as a data controller and/or processor, such as drawing up and implementing a data breach procedure, performing data-protection impact assessments, and developing a training plan.
|
||||
|
||||
The services include:
|
||||
* A dedicated data protection officer, available for email, telephone and video call advice.
|
||||
* Registration as DPO with the relevant supervisory authority.
|
||||
* Acting as the contact point with the relevant supervisory authority on all data protection matters.
|
||||
* GDPR compliance monitoring, which includes managing your GDPR/DPA 2018 compliance action plan.
|
||||
* A GDPR/DPA 2018 gap analysis and remedial action plan (year 1)
|
||||
* An annual compliance audit (from year 2).
|
||||
* Hands-on support with creating and maintaining your personal data processing register (Article 30 record).
|
||||
* Advising on data protection and maintaining compliance with the GDPR/DPA 2018.
|
||||
* Facilitating staff awareness training.
|
||||
* Support to identify personal data processing activities and verify that the data processing activities are GDPR compliant.
|
||||
* GDPR documentation review (policies and procedures), which includes a legal review for suitability and guidance on applicability.
|
||||
* Advice on handling DPIAs (data protection impact assessments), DSARs (data subject access requests), data breach monitoring, management, and reporting; and
|
||||
* Monthly activity reports and quarterly management reports.
|
||||
|
||||
[Toevoeging RK]:
|
||||
* Supporting reviews of existing and new Data Processing Agreements
|
||||
* Identifying GDPR compliance risk in your data processing activities and advising on risk reducing measures – dat valt impliciet onder DPIA’s maar misschien goed om het nog expliciet te benoemen
|
||||
|
||||
|
||||
The data protection officer shall in the performance of his or her tasks have due regard to the risk associated with processing operations, considering the nature, scope, context and purposes of processing.
|
||||
|
||||
DPOaaS is in principle an annual subscription product that is billed monthly.
|
||||
|
||||
Without knowing the exact scope and the level of actual compliance it is difficult to give a fixed quote but indicative we believe that we would need, for the preparation and set up 30 days and on an annual basis 120 days (10 days per months). In case of a three contract, we could spread the initial set up over the three years.
|
||||
|
|
@ -11,4 +11,4 @@ Additional resources and support are available on the iso27diy.com website.
|
|||
* No need for external consultants or expensive software
|
||||
|
||||
|
||||
See also [ISO27DIY benefits](../Sparks/iso27diy/ISO27DIY%20benefits.md)
|
||||
See also [ISO27DIY benefits](../../AuditGlue/ISO27DIY%20benefits.md)
|
||||
7
Corpus/Various/Fogg Behavior Model.md
Normal file
7
Corpus/Various/Fogg Behavior Model.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Fogg Behavior Model shows
|
||||
|
||||
https://behaviormodel.org
|
||||
|
||||
|
||||
The Fogg Behavior Model shows that **three elements must converge at the same moment for a behavior to occur: Motivation, Ability**, and a **Prompt**. When a behavior does not occur, at least one of those three elements is missing.
|
||||
|
||||
BIN
Corpus/Various/Fogg behaviour model chart.png
Normal file
BIN
Corpus/Various/Fogg behaviour model chart.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 307 KiB |
23
Corpus/Various/How Paperless.io attracted users.md
Normal file
23
Corpus/Various/How Paperless.io attracted users.md
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
Source: [Indie Hackers](https://www.indiehackers.com/post/on-building-growing-a-document-automation-saas-to-2k-mrr-8abc4b43d8?utm_campaign=post-8abc4b43d8&utm_medium=email&utm_source=ih-series-growth-acquisition)
|
||||
Interviewee: [Jacob Engels](https://www.linkedin.com/in/jacob-engels-1166b5146/)
|
||||
|
||||
## How have you attracted users and grown Paperless?
|
||||
|
||||
Our first channel was cold email outreach. We created a pretty incredible process of finding leads on LinkedIn, enriching data via a personal assistant and then personalizing outreach at scale. We had opening rates of up to 80%, reply rates of ~40-45% and this was just crazy. We also tested various industries and verticals to see where we could find the biggest traction.
|
||||
|
||||
In more detail:
|
||||
|
||||
We were using the LinkedIn Sales Navigator to build solid lists of interesting contacts. Trying different batches of leads (company size and industry) helped us come up with around 30 different target groups we were contacting.
|
||||
|
||||
Through Phantombuster we were exporting those contacts into an Excel File. Our personal assistant later searched for the email addresses with different tools such as [Hunter.io](https://click.pstmrk.it/2m/Hunter.io/pxBB3ScN/g09c/b1wYDVrXAZ/LU1yOEJIQ0N4MjlMa2ZJUzBJYW0 "https://click.pstmrk.it/2m/Hunter.io/pxBB3ScN/g09c/b1wYDVrXAZ/LU1yOEJIQ0N4MjlMa2ZJUzBJYW0") Rocketreach, Leadleaper, Name2email, SalesQl, Signal Hire, Snovio and Clearbit.
|
||||
|
||||
The final Leads + email addresses were exported to a sales automation tool named [Klenty.com](https://click.pstmrk.it/2m/Klenty.com/qBBB3ScN/g09c/oeoAOd4Ntu/LU1yOEJIQ0N4MjlMa2ZJUzBJYW0). WIth simple A/B/C Tests we were steadily finding out which headlines and what content prompts leads to answer our emails. Four simple rules helped us achieve those crazy high numbers:
|
||||
|
||||
KISS - Keep it simple and stupid.
|
||||
Nail the tone - Don’t smell like a salesman or marketeer
|
||||
Simple yes/no CTA
|
||||
Follow-Up x3
|
||||
|
||||
Depending on the answer and considering the Self-SignUp was not yet ready we were pushing hard for a 30 min call to generate insights and create a need/urgency. In the call, it was all about asking the right questions, but that’s worth another story :)
|
||||
|
||||
Initially, we let our early access users into our platform in exchange for feedback. Some of them converted to paid customers and we’re still heavily focused on interviewing and speaking with these early adopters to really nail our positioning, find product-market-fit and determine our best customers worth focusing on.
|
||||
9
Corpus/Various/LLM Security Models.md
Normal file
9
Corpus/Various/LLM Security Models.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# LLM's for the Security domain
|
||||
|
||||
[Basti8499/bge-large-en-v1.5-ISO-27001 · Hugging Face](https://huggingface.co/Basti8499/bge-large-en-v1.5-ISO-27001)
|
||||
https://huggingface.co/ZySec-AI/SecurityLLM
|
||||
|
||||
[Models](https://huggingface.co/models?other=cybersecurity&sort=likes)
|
||||
|
||||
[Google Gemini](https://gemini.google.com/u/2/app/44c7cb4ab371623a)
|
||||
[Google Gemini](https://gemini.google.com/app/0888b1e1426f1bf9)
|
||||
BIN
Corpus/Various/PANOPTIC-MTR.517c2bf5fd91f93354aa.pdf
Normal file
BIN
Corpus/Various/PANOPTIC-MTR.517c2bf5fd91f93354aa.pdf
Normal file
Binary file not shown.
BIN
Corpus/Various/PANOPTIC_version_1.b6b9783933c97d02395e.xlsx
Normal file
BIN
Corpus/Various/PANOPTIC_version_1.b6b9783933c97d02395e.xlsx
Normal file
Binary file not shown.
578
Corpus/Various/PDF-to-RAG pipeline.md
Normal file
578
Corpus/Various/PDF-to-RAG pipeline.md
Normal file
|
|
@ -0,0 +1,578 @@
|
|||
---
|
||||
tags:
|
||||
- llm
|
||||
- rag
|
||||
---
|
||||
|
||||
## Prompt
|
||||
```
|
||||
Advise on an architecture for a system that can do the following:
|
||||
|
||||
1) extract paragraphs from PDF and write them to a properly structured Markdown file, including tags
|
||||
2) rewrite the file in easily understandable and actionable English
|
||||
3) Add relevant content to the Markdown file through API calls to online LLM services
|
||||
4) Put the Markdown files into a database that can support a local LLM / RAG system
|
||||
|
||||
All components must be freely available and open source, and be able to run on an Apple Mac Pro M2 with 32 GB RAM.
|
||||
Preferably base it around LM Studio and/ or AnythingLLM.
|
||||
```
|
||||
|
||||
# Claude Sonnet 3.7
|
||||
|
||||
Here's a comprehensive architecture for your PDF-to-RAG pipeline using open-source components optimized for Mac M2:
|
||||
|
||||
## Core Architecture Overview
|
||||
|
||||
**Pipeline Flow**: PDF → Text Extraction → Markdown Generation → Content Enhancement → Database Storage → RAG System
|
||||
|
||||
## Component Stack
|
||||
|
||||
### 1. PDF Text Extraction
|
||||
**Primary Tool: PyMuPDF (fitz)**
|
||||
- Excellent paragraph detection and structure preservation
|
||||
- Fast performance on M2 architecture
|
||||
- Handles complex layouts and maintains formatting context
|
||||
|
||||
**Alternative: pdfplumber**
|
||||
- Better for tables and structured data
|
||||
- Can be used in conjunction with PyMuPDF
|
||||
|
||||
### 2. Local LLM Runtime
|
||||
**LM Studio** (as requested)
|
||||
- Excellent M2 optimization with Metal acceleration
|
||||
- Easy model management and API compatibility
|
||||
- Supports various model formats (GGUF, etc.)
|
||||
|
||||
**Recommended Models for 32GB RAM:**
|
||||
- Llama 3.1 8B or 13B variants
|
||||
- Mistral 7B models
|
||||
- Code Llama for technical content
|
||||
|
||||
### 3. Orchestration Framework
|
||||
**LangChain + Python**
|
||||
- Mature PDF processing chains
|
||||
- Built-in LLM integrations
|
||||
- Extensive documentation and community
|
||||
|
||||
**Core Script Structure:**
|
||||
```python
|
||||
# pdf_processor.py
|
||||
class PDFProcessor:
|
||||
def __init__(self):
|
||||
self.extractor = PyMuPDFExtractor()
|
||||
self.llm_client = LMStudioClient()
|
||||
self.db_client = ChromaDBClient()
|
||||
|
||||
def process_pipeline(self, pdf_path):
|
||||
# 1. Extract paragraphs
|
||||
paragraphs = self.extract_paragraphs(pdf_path)
|
||||
|
||||
# 2. Generate structured markdown
|
||||
markdown = self.generate_markdown(paragraphs)
|
||||
|
||||
# 3. Enhance with LLM
|
||||
enhanced_md = self.enhance_content(markdown)
|
||||
|
||||
# 4. Store in vector DB
|
||||
self.store_embeddings(enhanced_md)
|
||||
```
|
||||
|
||||
### 4. Vector Database
|
||||
**ChromaDB** (Recommended)
|
||||
- Lightweight and fast
|
||||
- Excellent Python integration
|
||||
- Runs efficiently on M2
|
||||
- Built-in embedding support
|
||||
|
||||
**Alternative: Qdrant**
|
||||
- More advanced filtering capabilities
|
||||
- Better for large-scale deployments
|
||||
|
||||
### 5. RAG System Integration
|
||||
**AnythingLLM** (as requested)
|
||||
- Direct ChromaDB integration
|
||||
- Web interface for document management
|
||||
- Supports custom embedding models
|
||||
|
||||
**Configuration:**
|
||||
- Connect AnythingLLM to your ChromaDB instance
|
||||
- Point to LM Studio API endpoint
|
||||
- Use local embedding models (e.g., all-MiniLM-L6-v2)
|
||||
|
||||
## Detailed Implementation Strategy
|
||||
|
||||
### Phase 1: PDF Processing Pipeline
|
||||
```python
|
||||
# Key libraries
|
||||
- PyMuPDF: PDF text extraction
|
||||
- python-frontmatter: Markdown metadata
|
||||
- requests: API calls to external LLMs
|
||||
- langchain: Orchestration
|
||||
```
|
||||
|
||||
### Phase 2: Content Enhancement
|
||||
**Local Processing:**
|
||||
- Use LM Studio API for rewriting and structuring
|
||||
- Implement prompt templates for consistency
|
||||
- Add retry logic for API reliability
|
||||
|
||||
**External API Integration:**
|
||||
- OpenRouter for accessing various models
|
||||
- Implement rate limiting and error handling
|
||||
- Cache responses to avoid redundant calls
|
||||
|
||||
### Phase 3: Database Architecture
|
||||
**ChromaDB Schema:**
|
||||
```python
|
||||
collection_config = {
|
||||
"name": "enhanced_documents",
|
||||
"metadata": {
|
||||
"source_file": "string",
|
||||
"page_number": "int",
|
||||
"section_type": "string",
|
||||
"enhancement_timestamp": "datetime"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: RAG Integration
|
||||
**AnythingLLM Setup:**
|
||||
- Configure workspace with ChromaDB backend
|
||||
- Set up document chunking strategy (512-1024 tokens)
|
||||
- Enable conversation memory for context retention
|
||||
|
||||
## Recommended File Structure
|
||||
```
|
||||
pdf-rag-system/
|
||||
├── src/
|
||||
│ ├── extractors/
|
||||
│ │ ├── pdf_extractor.py
|
||||
│ │ └── paragraph_detector.py
|
||||
│ ├── processors/
|
||||
│ │ ├── markdown_generator.py
|
||||
│ │ └── content_enhancer.py
|
||||
│ ├── storage/
|
||||
│ │ ├── chromadb_client.py
|
||||
│ │ └── embedding_manager.py
|
||||
│ └── api/
|
||||
│ ├── lm_studio_client.py
|
||||
│ └── external_llm_client.py
|
||||
├── config/
|
||||
│ ├── llm_prompts.yaml
|
||||
│ └── processing_config.yaml
|
||||
├── data/
|
||||
│ ├── input_pdfs/
|
||||
│ ├── processed_markdown/
|
||||
│ └── embeddings/
|
||||
└── docker-compose.yml (optional)
|
||||
```
|
||||
|
||||
## Performance Optimization for M2
|
||||
|
||||
**Memory Management:**
|
||||
- Process PDFs in batches to avoid memory overflow
|
||||
- Use streaming for large documents
|
||||
- Implement garbage collection between processing cycles
|
||||
|
||||
**M2-Specific Optimizations:**
|
||||
- Enable Metal acceleration in LM Studio
|
||||
- Use ARM64-optimized Python packages
|
||||
- Configure ChromaDB for Apple Silicon
|
||||
|
||||
## Integration Workflow
|
||||
|
||||
1. **Document Ingestion**: Drop PDFs into watched folder
|
||||
2. **Automated Processing**: Background service processes new files
|
||||
3. **Quality Control**: Manual review of enhanced markdown
|
||||
4. **Database Update**: Automatic embedding and storage
|
||||
5. **RAG Access**: Query through AnythingLLM interface
|
||||
|
||||
This architecture provides a scalable, maintainable system that leverages the M2's capabilities while staying within the open-source ecosystem. The modular design allows you to swap components as needed and provides clear upgrade paths for future enhancements.
|
||||
# Gemini PRO preview
|
||||
Here's a proposed architecture for your PDF processing and RAG system, focusing on open-source tools compatible with your Apple Mac Pro M2:
|
||||
This system is designed in a modular way, allowing you to process documents through a pipeline.
|
||||
## System Architecture Overview
|
||||
Code snippet
|
||||
```
|
||||
|
||||
graph TD
|
||||
A[PDF Files] --> B(1. PDF to Markdown Conversion);
|
||||
B -- Structured Markdown --> C(2. Content Rewriting - Local LLM);
|
||||
C -- Rewritten Markdown --> D(3. Content Enrichment - Online LLM);
|
||||
D -- Enriched Markdown --> E(4. Storage & RAG System);
|
||||
|
||||
subgraph "Component 1: PDF to Markdown"
|
||||
B_Tool[Python + marker-pdf library]
|
||||
end
|
||||
B --> B_Tool; B_Tool --> B;
|
||||
|
||||
|
||||
subgraph "Component 2: Rewriting (Local)"
|
||||
C_Script[Python Script]
|
||||
C_LLM_Server[LM Studio or Ollama]
|
||||
C_Model[Local LLM e.g., Gemma, Mistral - GGUF]
|
||||
end
|
||||
C --> C_Script; C_Script -- API Call --> C_LLM_Server; C_LLM_Server -- Hosts --> C_Model; C_LLM_Server --> C_Script;
|
||||
|
||||
subgraph "Component 3: Enrichment (Online)"
|
||||
D_Script[Python Script]
|
||||
D_Online_LLM[Online LLM Service API e.g., Perplexity, Groq]
|
||||
end
|
||||
D --> D_Script; D_Script -- API Call --> D_Online_LLM; D_Online_LLM --> D_Script;
|
||||
|
||||
subgraph "Component 4: RAG System"
|
||||
E_Interface[AnythingLLM]
|
||||
E_DB[LanceDB (managed by AnythingLLM)]
|
||||
E_Local_LLM[Local LLM via LM Studio/Ollama for RAG]
|
||||
end
|
||||
E --> E_Interface; E_Interface -- Manages --> E_DB; E_Interface -- Uses --> E_Local_LLM;
|
||||
|
||||
classDef component fill:#f9f,stroke:#333,stroke-width:2px;
|
||||
classDef tool fill:#lightgrey,stroke:#333,stroke-width:1px;
|
||||
|
||||
class A,B,C,D,E component;
|
||||
class B_Tool,C_Script,C_LLM_Server,C_Model,D_Script,D_Online_LLM,E_Interface,E_DB,E_Local_LLM tool;
|
||||
|
||||
```
|
||||
### 1. PDF to Markdown Extraction & Structuring
|
||||
* **Core Tool**: marker-pdf (Python library)
|
||||
* **Why**: marker-pdf is specifically designed to convert PDFs (including scanned ones via OCR) into Markdown with high accuracy. It uses deep learning models to understand document structure, identify paragraphs, headings, lists, tables, code blocks, and even convert equations to LaTeX. It can also extract images. This directly addresses your need for "properly structured Markdown."
|
||||
* **Process**:
|
||||
1 A Python script will use the marker-pdf library to process each input PDF.
|
||||
### 2 marker-pdf analyzes the PDF layout, extracts text content in reading order, and formats it as Markdown.
|
||||
3 "Tags" in this context will primarily be the structural elements inherent in Markdown (e.g., # H1, ## H2 for headings, * list item for lists, which marker-pdf generates). If you need semantic tags (keywords, topics), these can be added in a subsequent step using an LLM.
|
||||
* **Output**: A .md file for each PDF, with well-defined paragraphs and structure.
|
||||
* **Open Source & Mac Compatibility**: Yes, marker-pdf is open source and its dependencies (like PyTorch for deep learning models) support Apple Silicon (MPS for GPU acceleration).
|
||||
|
||||
⠀
|
||||
### 2. Rewrite in Understandable & Actionable English
|
||||
* **Core Tools**:
|
||||
* **Local LLM Runner**: **LM Studio** or **Ollama**.
|
||||
* **Why**: Both are excellent for downloading, managing, and running various open-source LLMs locally on your Mac. They provide an OpenAI-compatible API endpoint, making it easy to interact with the models programmatically.
|
||||
* **LLM Models**: Quantized versions (e.g., GGUF) of models like **Mistral 7B**, **Llama 3 8B**, **Gemma 7B**, or slightly larger models if your 32GB RAM allows comfortably alongside other processes.
|
||||
* **Why**: These models offer a good balance of performance and resource consumption. Quantization is key for fitting them into 32GB RAM.
|
||||
* **Control Script**: A Python script.
|
||||
* **Process**:
|
||||
1 The Python script reads the Markdown file generated in Step 1.
|
||||
2 It iterates through paragraphs or sections of the Markdown.
|
||||
3 For each piece of content, it sends a request to the local LLM API (e.g., LM Studio's server at http://localhost:1234/v1/chat/completions).
|
||||
4 The prompt would be engineered for the task, e.g., "Rewrite the following text to be easily understandable and actionable for a general audience. Focus on clarity and conciseness:\n\n[paragraph_text]".
|
||||
5 The script receives the rewritten text from the LLM and updates the Markdown file.
|
||||
* **Output**: The original Markdown file, now with its content rewritten.
|
||||
|
||||
⠀
|
||||
### 3. Add Relevant Content via Online LLM Services
|
||||
* **Core Tools**:
|
||||
* **API Interaction**: Python script using the requests library (standard) or a dedicated library like llm by Datasette if it simplifies connections to multiple services.
|
||||
* **Online LLM Services**: Choose services that offer free tiers or APIs you have access to (e.g., Groq API for fast inference, Perplexity API for search-augmented generation, or others). Ensure their terms align with your use case.
|
||||
* **Process**:
|
||||
1 The Python script parses the rewritten Markdown file.
|
||||
2 It identifies key topics, entities, or questions within the text that could be expanded upon. (This could be heuristic-based, e.g., looking for specific keywords, or you could even use the local LLM from Step 2 to suggest areas for enrichment).
|
||||
3 The script formulates prompts for the chosen online LLM API (e.g., "Provide a concise summary of recent advancements in [topic from Markdown]" or "Explain the concept of [term from Markdown] in simple terms").
|
||||
4 It makes the API call, retrieves the information.
|
||||
5 The new content is intelligently integrated into the Markdown file (e.g., as a new subsection, an appended note, or an in-text citation/explanation).
|
||||
* **Output**: The Markdown file, now enriched with additional relevant information.
|
||||
|
||||
⠀
|
||||
### 4. Store Markdown for Local LLM / RAG System
|
||||
* **Core Tools**:
|
||||
* **Primary RAG Platform**: **AnythingLLM**
|
||||
* **Why**: AnythingLLM is an open-source, privacy-focused application designed specifically for building RAG systems. It runs locally, supports various LLM backends (including LM Studio and Ollama), and uses **LanceDB** as its default vector database. LanceDB is a modern, embedded, serverless vector database that is efficient and requires no separate setup. AnythingLLM handles document ingestion (including Markdown), chunking, embedding, and provides a chat interface.
|
||||
* **Vector Database (managed by AnythingLLM)**: **LanceDB**
|
||||
* **Local LLM for RAG**: The same LLMs used in Step 2, run via LM Studio or Ollama, can be configured as the chat/reasoning model within AnythingLLM.
|
||||
* **Process**:
|
||||
**1** **Setup AnythingLLM**: Install and configure AnythingLLM on your Mac.
|
||||
**2** **Configure LLM**: Point AnythingLLM to your local LLM server (e.g., LM Studio's API endpoint). Select an embedding model within AnythingLLM (many SBERT-like models can run locally and efficiently).
|
||||
**3** **Ingest Markdown**: Upload the final, enriched Markdown files from Step 3 directly into an AnythingLLM "workspace."
|
||||
* AnythingLLM will automatically:
|
||||
* Chunk the Markdown content.
|
||||
* Generate embeddings for each chunk using the selected embedding model.
|
||||
* Store the chunks and their embeddings in its LanceDB instance.
|
||||
**4** **Chat & Retrieve**: Use the AnythingLLM interface to ask questions or interact with your documents. AnythingLLM will perform similarity searches in LanceDB to find relevant chunks (retrieval) and then pass these chunks along with your query to the configured local LLM to generate an answer (augmented generation).
|
||||
* **Output**: A fully functional RAG system allowing you to query and get insights from your processed PDF content.
|
||||
|
||||
⠀
|
||||
### Workflow & Data Flow
|
||||
1. PDFs are fed into a Python script utilizing marker-pdf.
|
||||
2. marker-pdf outputs **structured Markdown files**.
|
||||
3. Another Python script takes these Markdown files, sends content paragraph by paragraph to a **local LLM via LM Studio/Ollama** for rewriting.
|
||||
4. The **rewritten Markdown files** are then processed by a third Python script, which identifies areas for enrichment and calls **online LLM APIs**.
|
||||
5. The **enriched Markdown files** are the final documents.
|
||||
6. These Markdown files are uploaded into **AnythingLLM**, which processes them into its **LanceDB** vector store.
|
||||
7. Users interact with their data through the **AnythingLLM interface**, which uses a **local LLM (via LM Studio/Ollama)** for generating responses based on retrieved context.
|
||||
|
||||
⠀Running on Apple Mac Pro M2 (32 GB RAM)
|
||||
* marker-pdf: Leverages PyTorch, which can use Metal Performance Shaders (MPS) on your M2 for hardware acceleration.
|
||||
* **Local LLMs**: Crucially, use quantized models (e.g., 4-bit or 5-bit GGUF/AWQ/GPTQ). LM Studio and Ollama make this easy. Your 32GB RAM is sufficient for running powerful models (e.g., 7B to potentially 30B+ quantized models, depending on their exact size and other running processes). Monitor RAM usage.
|
||||
* **AnythingLLM & LanceDB**: Designed to be efficient and run locally. LanceDB is embedded, reducing overhead.
|
||||
* **Concurrency**: You'll be running multiple components, potentially simultaneously (e.g., LM Studio server, AnythingLLM, Python scripts). 32GB RAM should handle this, but be mindful of the number of LLMs loaded at once and their size.
|
||||
|
||||
This architecture provides a robust, open-source pipeline that should meet all your requirements and run effectively on your specified hardware. Remember to install Python and the necessary libraries (marker-pdf, requests, ollama or manage models through LM Studio GUI).
|
||||
|
||||
# Perplexity
|
||||
Here's an open-source architecture for your PDF-to-RAG system optimized for Apple Silicon:
|
||||
## PDF Extraction & Markdown Conversion
|
||||
### **1** **PyMuPDF (fitz)**
|
||||
* Use `page.get_text("blocks")` to extract paragraph chunks
|
||||
* Handles complex layouts better than PDFMiner/PyPDF2 ~[2](https://www.reddit.com/r/LangChain/comments/1e7cntq/whats_the_best_python_library_for_extracting_text/)~
|
||||
* Alternative: **Marker** for GPU-accelerated conversion (CPU-only mode works on Mac) ~[1](https://github.com/VikParuchuri/marker)~
|
||||
|
||||
```python
|
||||
import fitz
|
||||
|
||||
def extract_paragraphs(pdf_path):
|
||||
doc = fitz.open(pdf_path)
|
||||
paragraphs = []
|
||||
for page in doc:
|
||||
blocks = page.get_text("blocks")
|
||||
paragraphs.extend([b[4] for b in blocks if b[6] == 0]) # Filter text blocks
|
||||
return paragraphs
|
||||
```
|
||||
## Text Rewriting & Enhancement
|
||||
### **2** **Mistral-7B via LM Studio**
|
||||
* Local inference with 4-bit quantization (fits in 32GB RAM)
|
||||
* Prompt template for simplification:
|
||||
```text
|
||||
Rewrite this technical content in simple, actionable English.
|
||||
Maintain original structure and key terms. Output MARKDOWN.
|
||||
{input_text}
|
||||
```
|
||||
### **3** **API Integration Layer**
|
||||
Use **FastAPI** to create endpoints for:
|
||||
* Jina Reader API (free tier) for web content enrichment ~[5](https://jina.ai/reader/)~
|
||||
* Firecrawl OSS for website crawling ~[4](https://www.reddit.com/r/mlops/comments/1c5usna/an_oss_tool_for_turning_entire_websites_into/)~
|
||||
|
||||
## Database & RAG Integration
|
||||
### **4** **AnythingLLM with ChromaDB**
|
||||
* Preconfigured ingestion pipeline for Markdown files
|
||||
* Supports local embeddings with **all-MiniLM-L6-v2**
|
||||
* Custom metadata tagging:
|
||||
```yaml
|
||||
processing:
|
||||
- step: add_frontmatter
|
||||
params:
|
||||
tags: ["technical", "user_manual"]
|
||||
```
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[PDF Input] --> B(PyMuPDF/Marker)
|
||||
B --> C[Raw Markdown]
|
||||
C --> D{Mistral-7B\nLM Studio}
|
||||
D --> E[Simplified MD]
|
||||
E --> F[API Enrichment]
|
||||
F --> G[AnythingLLM]
|
||||
G --> H[ChromaDB]
|
||||
H --> I[RAG System]
|
||||
```
|
||||
|
||||
## Recommended Stack
|
||||
| **Component** | **Technology** | **Resources** |
|
||||
|:-:|:-:|:-:|
|
||||
| PDF Processing | PyMuPDF + Marker | ~[GitHub: pymupdf](https://github.com/pymupdf/PyMuPDF)~ |
|
||||
| Local LLM | Mistral-7B | ~[LM Studio](https://lmstudio.ai/)~ |
|
||||
| RAG System | AnythingLLM | ~[GitHub: anythingllm](https://github.com/Mintplex-Labs/anything-llm)~ |
|
||||
| Vector Store | ChromaDB | ~[Chroma Docs](https://docs.trychroma.com/)~ |
|
||||
|
||||
All components are MIT/Apache-licensed and optimized for Apple Silicon acceleration. The system can handle ~50-page PDFs in <30s on M2 Pro hardware~[2](https://www.reddit.com/r/LangChain/comments/1e7cntq/whats_the_best_python_library_for_extracting_text/)[4](https://www.reddit.com/r/mlops/comments/1c5usna/an_oss_tool_for_turning_entire_websites_into/)~. For production scaling, consider adding Celery for async task queues.
|
||||
## Citations
|
||||
1 ~[https://github.com/VikParuchuri/marker](https://github.com/VikParuchuri/marker)~
|
||||
2 ~[https://www.reddit.com/r/LangChain/comments/1e7cntq/whats_the_best_python_library_for_extracting_text/](https://www.reddit.com/r/LangChain/comments/1e7cntq/whats_the_best_python_library_for_extracting_text/)~
|
||||
3 ~[https://github.com/mundimark/awesome-markdown-editors](https://github.com/mundimark/awesome-markdown-editors)~
|
||||
4 ~[https://www.reddit.com/r/mlops/comments/1c5usna/an_oss_tool_for_turning_entire_websites_into/](https://www.reddit.com/r/mlops/comments/1c5usna/an_oss_tool_for_turning_entire_websites_into/)~
|
||||
5 ~[https://jina.ai/reader/](https://jina.ai/reader/)~
|
||||
6 ~[https://github.com/Karthick-840/Crawl4ai-RAG-with-Local-LLM](https://github.com/Karthick-840/Crawl4ai-RAG-with-Local-LLM)~
|
||||
7 ~[https://github.com/kevwan/rag-agent](https://github.com/kevwan/rag-agent)~
|
||||
8 ~[https://blog.stephenturner.us/p/gui-local-llm-rag](https://blog.stephenturner.us/p/gui-local-llm-rag)~
|
||||
9 ~[https://github.com/opendatalab/MinerU](https://github.com/opendatalab/MinerU)~
|
||||
10 ~[https://opensource.com/article/18/11/markdown-editors](https://opensource.com/article/18/11/markdown-editors)~
|
||||
11 ~[https://www.reddit.com/r/LangChain/comments/1fsd1yw/best_open_source_rag_for_100s_of_pdfs/](https://www.reddit.com/r/LangChain/comments/1fsd1yw/best_open_source_rag_for_100s_of_pdfs/)~
|
||||
12 ~[https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/](https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/)~
|
||||
13 ~[https://www.reddit.com/r/MachineLearning/comments/1hg5d3p/p_vision_parse_parse_pdf_documents_into_markdown/](https://www.reddit.com/r/MachineLearning/comments/1hg5d3p/p_vision_parse_parse_pdf_documents_into_markdown/)~
|
||||
14 ~[https://www.reddit.com/r/ObsidianMD/comments/1hioaov/microsoft_has_released_an_open_source_python_tool/](https://www.reddit.com/r/ObsidianMD/comments/1hioaov/microsoft_has_released_an_open_source_python_tool/)~
|
||||
15 ~[https://www.matillion.com/blog/step-by-step-guide-building-a-rag-model-with-open-source-llm-llama-2-and-vector-store-faiss](https://www.matillion.com/blog/step-by-step-guide-building-a-rag-model-with-open-source-llm-llama-2-and-vector-store-faiss)~
|
||||
16 ~[https://pymupdf.readthedocs.io/en/latest/pymupdf4llm](https://pymupdf.readthedocs.io/en/latest/pymupdf4llm)~
|
||||
17 ~[https://discourse.devontechnologies.com/t/the-open-source-project-mineru-is-highly-recommended-as-a-tool-for-pdf-to-markdown/83016](https://discourse.devontechnologies.com/t/the-open-source-project-mineru-is-highly-recommended-as-a-tool-for-pdf-to-markdown/83016)~
|
||||
18 ~[https://www.nutrient.io/blog/how-to-extract-text-from-a-pdf-using-javascript/](https://www.nutrient.io/blog/how-to-extract-text-from-a-pdf-using-javascript/)~
|
||||
19 ~[https://edrawmind.wondershare.com/productivity-improvement/8-best-open-source-markdown-editors.html](https://edrawmind.wondershare.com/productivity-improvement/8-best-open-source-markdown-editors.html)~
|
||||
20 ~[https://github.com/h7ml/ai-markdown-translator](https://github.com/h7ml/ai-markdown-translator)~
|
||||
21 ~[https://scrapingant.com/blog/markdown-efficient-data-extraction](https://scrapingant.com/blog/markdown-efficient-data-extraction)~
|
||||
22 ~[https://apidog.com/blog/free-open-source-llm-apis/](https://apidog.com/blog/free-open-source-llm-apis/)~
|
||||
23 ~[https://blog.ovhcloud.com/reference-architecture-retrieval-augmented-generation-rag/](https://blog.ovhcloud.com/reference-architecture-retrieval-augmented-generation-rag/)~
|
||||
24 ~[https://github.com/phmullins/awesome-macos](https://github.com/phmullins/awesome-macos)~
|
||||
25 ~[https://anythingllm.com](https://anythingllm.com/)~
|
||||
26 ~[https://digitaconnect.com/how-to-implement-rag-using-anythingllm-and-lm-studio/](https://digitaconnect.com/how-to-implement-rag-using-anythingllm-and-lm-studio/)~
|
||||
27 ~[https://www.youtube.com/watch?v=7atkVfm3LyY](https://www.youtube.com/watch?v=7atkVfm3LyY)~
|
||||
28 ~[https://ai.gopubby.com/benchmarking-pdf-to-markdown-document-converters-fc65a2c73bf2](https://ai.gopubby.com/benchmarking-pdf-to-markdown-document-converters-fc65a2c73bf2)~
|
||||
29 ~[https://notegpt.io/pdf-to-markdown-converter](https://notegpt.io/pdf-to-markdown-converter)~
|
||||
30 ~[https://dillinger.io](https://dillinger.io/)~
|
||||
31 ~[https://markdowntotext.com](https://markdowntotext.com/)~
|
||||
32 ~[https://portalzine.de/the-best-open-source-markdown-editors-for-writing-and-beyond/](https://portalzine.de/the-best-open-source-markdown-editors-for-writing-and-beyond/)~
|
||||
33 ~[https://www.andrewheiss.com/blog/2019/10/09/convert-md-rtf-macos-services/](https://www.andrewheiss.com/blog/2019/10/09/convert-md-rtf-macos-services/)~
|
||||
34 ~[https://stackedit.io](https://stackedit.io/)~
|
||||
35 ~[https://github.com/supermemoryai/markdowner](https://github.com/supermemoryai/markdowner)~
|
||||
36 ~[https://github.com/Jing-yilin/E2M](https://github.com/Jing-yilin/E2M)~
|
||||
37 ~[https://llm.md](https://llm.md/)~
|
||||
38 ~[https://blog.stephenturner.us/p/turn-any-webpage-into-markdown-for-llm-friendly-input](https://blog.stephenturner.us/p/turn-any-webpage-into-markdown-for-llm-friendly-input)~
|
||||
39 ~[https://apify.com/easyapi/website-content-to-markdown-for-llm-training](https://apify.com/easyapi/website-content-to-markdown-for-llm-training)~
|
||||
40 ~[https://serpapi.com/blog/turning-search-results-into-markdown-for-llms/](https://serpapi.com/blog/turning-search-results-into-markdown-for-llms/)~
|
||||
41 ~[https://github.com/infiniflow/ragflow](https://github.com/infiniflow/ragflow)~
|
||||
42 ~[https://hub.athina.ai/top-5-open-source-scraping-and-ingestion-tools/](https://hub.athina.ai/top-5-open-source-scraping-and-ingestion-tools/)~
|
||||
43 ~[https://www.tensorops.ai/post/no-clouds-allowed-building-an-all-open-source-local-rag-system](https://www.tensorops.ai/post/no-clouds-allowed-building-an-all-open-source-local-rag-system)~
|
||||
44 ~[https://setapp.com/how-to/dbms-for-mac](https://setapp.com/how-to/dbms-for-mac)~
|
||||
45 ~[https://www.youtube.com/watch?v=4Kcm9Tsojjc](https://www.youtube.com/watch?v=4Kcm9Tsojjc)~
|
||||
46 ~[https://www.reddit.com/r/LocalLLaMA/comments/1c3zney/can_anyone_explain_to_me_how_to_create_a_rag_for/](https://www.reddit.com/r/LocalLLaMA/comments/1c3zney/can_anyone_explain_to_me_how_to_create_a_rag_for/)~
|
||||
47 ~[https://lmstudio.ai/docs/advanced/tool-use](https://lmstudio.ai/docs/advanced/tool-use)~
|
||||
48 ~[https://github.com/Mintplex-Labs/anything-llm](https://github.com/Mintplex-Labs/anything-llm)~
|
||||
49 ~[https://www.youtube.com/watch?v=bGBJhkZfDSY](https://www.youtube.com/watch?v=bGBJhkZfDSY)~
|
||||
50 ~[https://www.youtube.com/watch?v=Tc9eApj0PE4](https://www.youtube.com/watch?v=Tc9eApj0PE4)~
|
||||
|
||||
⠀
|
||||
Answer from Perplexity: ~[https://www.perplexity.ai/search/213dad02-2d0c-4700-8d4a-b9ec86a77a6f?utm_source=copy_output](https://www.perplexity.ai/search/213dad02-2d0c-4700-8d4a-b9ec86a77a6f?utm_source=copy_output)~
|
||||
|
||||
|
||||
# Gemini 2
|
||||
|
||||
## PDF to RAG
|
||||
https://gemini.google.com/app/f2fa5832739af240
|
||||
|
||||
This architecture outlines a system for processing PDFs, enhancing their content, and making them queryable via a local LLM/RAG system, all with freely available and open-source components runnable on an Apple Mac Pro M2 with 32 GB RAM.
|
||||
|
||||
## System Architecture
|
||||
|
||||
The system can be broken down into several sequential and interconnected stages:
|
||||
|
||||
**1. PDF Extraction and Markdown Structuring**
|
||||
|
||||
- **Tool:** `PyMuPDF4LLM` (Python Library)
|
||||
- **Functionality:** This library is designed specifically for extracting content from PDFs into Markdown format, making it suitable for LLM and RAG environments. It can detect standard text, tables, and headers, and convert them into GitHub-compatible Markdown with appropriate `#` tags for headings.
|
||||
- **Implementation:** A Python script will utilize `PyMuPDF4LLM` to process input PDFs.
|
||||
- **Output:** Structured Markdown files, where paragraphs are clearly defined, and headers are tagged.
|
||||
|
||||
**2. Content Rewriting for Clarity**
|
||||
|
||||
- **Tool:** Local LLM via `LM Studio` or `Ollama`
|
||||
- **Functionality:** Open-source LLMs can be used to rewrite the extracted Markdown content into easily understandable and actionable English. You'll need an LLM specifically trained or instruction-tuned for text summarization, simplification, and clarity.
|
||||
- **Model Selection:** Look for models on Hugging Face (e.g., from the Mistral, Llama, or Gemma families) that are instruction-tuned for tasks like "summarize," "rewrite for clarity," or "explain simply." Models with a `.gguf` extension are generally compatible with LM Studio and Ollama.
|
||||
- **Implementation:**
|
||||
- **LM Studio:** LM Studio provides a user-friendly GUI for downloading and running local LLMs. It also exposes a local server that can be used via an OpenAI-like API. Your Python script can interact with this local API to send Markdown content for rewriting.
|
||||
- **Ollama:** Ollama is another excellent choice for running LLMs locally via a command-line interface. It also provides an API for programmatic interaction.
|
||||
- **Process:** The Python script will read the initial Markdown file, send sections or the entire content to the local LLM with a specific prompt (e.g., "Rewrite this text for clarity and actionability."), and then update the Markdown file with the rewritten content.
|
||||
|
||||
**3. Content Augmentation via Online LLM Services**
|
||||
|
||||
- **Tool:** Any online LLM service with an API (e.g., OpenAI, Anthropic, Google Gemini API, etc.)
|
||||
- **Functionality:** For adding relevant content that might require broader knowledge or more complex reasoning, an online LLM API can be leveraged. This is useful for filling in gaps, providing context, or generating examples not present in the original PDF.
|
||||
- **Considerations:** While the core system is open-source, this step introduces a dependency on a paid service. However, it's the most flexible way to add external, high-quality content. You could potentially substitute this with an even larger local LLM if your Mac Pro M2 can handle it, but online services often have access to more recent or diverse training data.
|
||||
- **Implementation:** The Python script will make API calls to the chosen online LLM, passing specific queries or context derived from the Markdown file. The generated content will then be appended or integrated into the Markdown file.
|
||||
|
||||
**4. Database for Local LLM / RAG System**
|
||||
|
||||
- **RAG System Core:** `AnythingLLM` or `LM Studio`'s built-in RAG features
|
||||
- **Functionality:** Both AnythingLLM and LM Studio offer RAG capabilities. A RAG system allows your LLM to retrieve relevant information from your documents before generating a response, drastically reducing hallucinations and improving accuracy.
|
||||
- **Vector Database:** To enable RAG, your Markdown files need to be converted into numerical representations called "embeddings" and stored in a vector database.
|
||||
- **AnythingLLM:** AnythingLLM has built-in support for various embedding models and can manage the vector database internally. It often defaults to using a local vector database like `ChromaDB` or `LanceDB`.
|
||||
- **LM Studio:** LM Studio also has "Chat with Documents" functionality, which acts as a RAG system. It automatically handles the retrieval of relevant document chunks.
|
||||
- **Manual Setup (for more control):** If you desire more control, you could use Python libraries like `LangChain` or `LlamaIndex` to build a RAG pipeline. These frameworks integrate with various vector databases (e.g., `ChromaDB`, `FAISS`, `Milvus` - the last two are good for local use and are supported by `kevwan/rag-agent` on GitHub) and embedding models (e.g., `Sentence-Transformers`).
|
||||
- **Embedding Models:** For local embeddings, `nomic-embed-text` (available via Ollama) or models from the `Sentence-Transformers` library are good open-source choices.
|
||||
- **Implementation:**
|
||||
- **AnythingLLM:** Set up AnythingLLM, and upload your Markdown files to it. AnythingLLM will handle the embedding and indexing.
|
||||
- **LM Studio:** Utilize LM Studio's "Chat with Documents" feature by attaching the Markdown files to your chat sessions.
|
||||
- **Manual RAG:**
|
||||
- A Python script reads the processed Markdown files.
|
||||
- Uses an embedding model (e.g., from `Sentence-Transformers`) to create embeddings for each paragraph or chunk of text.
|
||||
- Stores these embeddings, along with references to the original Markdown content, in a local vector database (e.g., `ChromaDB`).
|
||||
- Integrates with `LM Studio` or `Ollama` as the LLM backend for queries. When a user asks a question, the system performs a similarity search in the vector database, retrieves relevant Markdown chunks, and then feeds these chunks as context to the LLM for generating a coherent answer.
|
||||
|
||||
## Workflow Diagram
|
||||
|
||||
Code snippet
|
||||
|
||||
```
|
||||
graph TD
|
||||
A[PDF Files] --> B{Python Script: PyMuPDF4LLM};
|
||||
B --> C[Structured Markdown Files];
|
||||
C --> D{Python Script: Local LLM Rewriting};
|
||||
D --> E[Rewritten Markdown Files];
|
||||
E --> F{Python Script: Online LLM Augmentation (Optional)};
|
||||
F --> G[Enhanced Markdown Files];
|
||||
G --> H{Vector Database Ingestion (AnythingLLM / LM Studio / Manual RAG)};
|
||||
H --> I[Vector Database (e.g., ChromaDB)];
|
||||
J[User Query] --> K{RAG System (AnythingLLM / LM Studio / LangChain/LlamaIndex)};
|
||||
K --> I;
|
||||
K --> L[Local LLM (LM Studio / Ollama)];
|
||||
L --> M[Generated Answer];
|
||||
```
|
||||
|
||||
## Detailed Component Breakdown & Considerations
|
||||
|
||||
### 1. PDF Extraction and Markdown Structuring
|
||||
|
||||
- **PyMuPDF4LLM:**
|
||||
- **Installation:** `pip install pymupdf4llm`
|
||||
- **Usage:** Simple Python API. You can loop through a directory of PDFs and convert each.
|
||||
- **Tagging:** `PyMuPDF4LLM` automatically identifies headers and adds Markdown tags (`#`, `##`, etc.). For paragraph-level tags, you might need to implement additional logic within your Python script based on keywords or other content patterns within the extracted text. This can be done by parsing the `md_text` output and inserting custom Markdown tags (e.g., `[tag:important]`, `[category:finance]`).
|
||||
|
||||
### 2. Content Rewriting
|
||||
|
||||
- **LM Studio / Ollama:**
|
||||
- **Installation:** Download the desktop app for LM Studio or use the `curl` command for Ollama on Mac.
|
||||
- **Model Download:** Within LM Studio's interface, you can search and download `.gguf` models. For Ollama, use `ollama pull <model_name>`.
|
||||
- **Model Choice for Rewriting:** Look for models that are good at instruction following and text generation. `Mistral-7B-Instruct-v0.2.Q5_K_M.gguf` or `Llama-3-8B-Instruct-Q5_K_M.gguf` are good starting points for 32GB RAM. Experiment with different quantizations (e.g., Q4, Q5) to find the best balance of performance and quality.
|
||||
- **API Interaction:** Both LM Studio and Ollama expose an OpenAI-compatible API endpoint (typically `http://localhost:1234/v1`). You can use the `openai` Python client library to interact with it, just pointing it to your local server.
|
||||
|
||||
### 3. Content Augmentation (Optional but Recommended)
|
||||
|
||||
- **Online LLM API:**
|
||||
- **Choice:** OpenAI's GPT models, Google's Gemini API, Anthropic's Claude, etc.
|
||||
- **Implementation:** Use the respective Python client libraries. You'll need API keys.
|
||||
- **Strategy:** Define specific triggers or conditions for when to call the online LLM (e.g., if a certain keyword is present, or if more detailed explanations are required for a particular topic).
|
||||
|
||||
### 4. Database for Local LLM / RAG System
|
||||
|
||||
- **AnythingLLM:**
|
||||
- **Installation:** Download the desktop application or use Docker (if you're comfortable with Docker).
|
||||
- **Setup:** Once installed, you can configure your LLM provider (point it to your local LM Studio/Ollama endpoint), embedding model, and then upload your Markdown files. AnythingLLM provides a web interface for managing your documents and chatting.
|
||||
- **Advantages:** User-friendly GUI, easy document management, built-in RAG.
|
||||
- **LM Studio (Built-in RAG):**
|
||||
- **Usage:** Simply drag and drop your Markdown files into the chat interface or use the "Chat with Documents" feature. LM Studio will automatically chunk and embed the documents for RAG.
|
||||
- **Advantages:** Extremely simple to set up if you're already using LM Studio for local LLMs.
|
||||
- **Manual RAG with LangChain/LlamaIndex + ChromaDB/FAISS:**
|
||||
- **LangChain/LlamaIndex:** Python frameworks for building LLM applications. They provide abstractions for document loading, splitting, embedding, vector store interaction, and RAG chains.
|
||||
- **ChromaDB:** A lightweight, open-source vector database that can run entirely in-memory or persist to disk. It's often used locally.
|
||||
- **Installation:** `pip install chromadb`
|
||||
- **FAISS:** Facebook AI Similarity Search, a library for efficient similarity search and clustering of dense vectors. Good for local, in-memory use.
|
||||
- **Installation:** `pip install faiss-cpu`
|
||||
- **Embedding Models:** Use `Sentence-Transformers` (e.g., `all-MiniLM-L6-v2`) or `OllamaEmbeddings` if using Ollama.
|
||||
- **Process:**
|
||||
1. **Load:** Use `MarkdownLoader` (LangChain) or `MarkdownReader` (LlamaIndex) to load your Markdown files.
|
||||
2. **Split:** Chunk the documents into smaller, manageable pieces (e.g., using `RecursiveCharacterTextSplitter`).
|
||||
3. **Embed:** Generate embeddings for each chunk using your chosen embedding model.
|
||||
4. **Store:** Store the chunks and their embeddings in ChromaDB or FAISS.
|
||||
5. **Retrieve & Generate:** When a query comes in, perform a similarity search on the vector database, retrieve the top-k relevant chunks, and then pass these chunks as context to your local LLM (running via LM Studio/Ollama API) to generate the final answer.
|
||||
|
||||
## System Requirements and Performance on Mac Pro M2 (32 GB RAM)
|
||||
|
||||
- **M2 Chip:** The Apple M2 chip, with its unified memory architecture and neural engine, is highly efficient for running local LLMs.
|
||||
- **32 GB RAM:** This amount of RAM is decent for running medium-sized LLMs (e.g., 7B-13B parameter models, especially when quantized to Q4 or Q5). You might even be able to experiment with larger models (e.g., 20B) if they are highly quantized, but performance might degrade.
|
||||
- **Performance:**
|
||||
- **PDF Extraction:** `PyMuPDF4LLM` is very fast and efficient.
|
||||
- **LLM Inference (Local):** LM Studio and Ollama are optimized for Apple Silicon. You should experience good inference speeds for rewriting and RAG queries, especially with quantized models.
|
||||
- **Embedding Generation:** Generating embeddings can be computationally intensive for a large number of documents, but the M2 chip will handle it much faster than a CPU-only machine.
|
||||
- **Vector Database:** ChromaDB and FAISS are efficient for local use.
|
||||
|
||||
## Recommended Setup Path
|
||||
|
||||
1. **Start with the Core:** Begin with `PyMuPDF4LLM` for PDF to Markdown conversion.
|
||||
2. **Local LLM with LM Studio/Ollama:** Get LM Studio or Ollama running and download a suitable instruction-tuned LLM. Test the rewriting capabilities manually first.
|
||||
3. **Automate Rewriting:** Write the Python script to connect `PyMuPDF4LLM` output to your local LLM for rewriting.
|
||||
4. **Implement RAG:**
|
||||
- **Simplest:** Use AnythingLLM or LM Studio's built-in RAG features. This will get you up and running quickly with minimal coding.
|
||||
- **More Control:** If you need more customization, delve into `LangChain` or `LlamaIndex` with `ChromaDB`.
|
||||
5. **Optional Augmentation:** If needed, integrate the online LLM API for advanced content generation.
|
||||
|
||||
This modular approach allows you to build and test each component independently, ensuring a robust and scalable system.
|
||||
|
||||
|
||||
BIN
Corpus/Various/PRAGMATIC_security_metrics_examples.xlsx
Normal file
BIN
Corpus/Various/PRAGMATIC_security_metrics_examples.xlsx
Normal file
Binary file not shown.
6
Corpus/Various/Privacy Threat Modeling.md
Normal file
6
Corpus/Various/Privacy Threat Modeling.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
MITRE PANOPTIC Privacy Threat Model
|
||||
- https://ptmworkshop.gitlab.io/#/panoptic
|
||||
- https://ptmworkshop.gitlab.io/static/media/PANOPTIC-MTR.517c2bf5fd91f93354aa.pdf
|
||||
|
||||
|
||||
|
||||
21
Corpus/Various/Rules to get better code.md
Normal file
21
Corpus/Various/Rules to get better code.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
---
|
||||
tags:
|
||||
- vibecoding
|
||||
---
|
||||
# Rules I give Claude to get better code (curious what works for you)
|
||||
|
||||
After months working with Claude for dev work, I built a set of strict instructions to avoid bad outputs, hallucinated code, or bloated files.
|
||||
|
||||
These rules consistently give me cleaner results, feel free to copy/adapt:
|
||||
|
||||
1. No artifacts.
|
||||
2. Less code is better than more code.
|
||||
3. No fallback mechanisms — they hide real failures.
|
||||
4. Rewrite existing components over adding new ones.
|
||||
5. Flag obsolete files to keep the codebase lightweight.
|
||||
6. Avoid race conditions at all costs.
|
||||
7. Always output the full component unless told otherwise.
|
||||
8. Never say “X remains unchanged” — always show the code.
|
||||
9. Be explicit on where snippets go (e.g., below “abc”, above “xyz”).
|
||||
10. If only one function changes, just show that one.
|
||||
11. Take your time to ultrathink when on extended thinking mode — thinking is cheaper than fixing bugs.
|
||||
31
Corpus/Various/Secure Vibecoding.md
Normal file
31
Corpus/Various/Secure Vibecoding.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
---
|
||||
tags:
|
||||
- dev
|
||||
- vibecoding
|
||||
- appsec
|
||||
---
|
||||
[Source](https://www.reddit.com/r/vibecoding/s/MoWTv0TLhQ)
|
||||
|
||||
**Here are some tips that i follow and might help people reading:**
|
||||
|
||||
1. Lockdown your backend (Supabase policies can help):
|
||||
|
||||
Most vibe-coded apps using Supabase or Firebase leave their backend wide open. Anyone who knows your endpoint URL can potentially view or modify sensitive data, like user accounts, subscriptions, or even payment info.
|
||||
|
||||
**What to do:** Don’t rely on default settings. Go into your Supabase project, open the **Auth Policies**, and restrict everything. By default, deny all access, and only allow specific users to access their own data.
|
||||
|
||||
**Why:** Even if your frontend looks secure, if your backend allows anyone to hit the database directly, you’re not just vulnerable, you’re exposed.
|
||||
|
||||
Resource: [Supabase RLS Docs ](https://supabase.com/docs/guides/database/postgres/row-level-security)
|
||||
|
||||
2. Don’t trust the frontend and always validate requests:
|
||||
Tools like Lovable or Bolt often generate frontend-heavy apps, where important actions (like account upgrades or profile edits) happen purely in the UI, with little to no checks behind the scenes.
|
||||
|
||||
**What to do:** Always assume that **anyone** can inspect, modify, and resend requests. Validate every request on the backend: check if the user is logged in, if they have the right role, and if they’re even allowed to touch that data.
|
||||
|
||||
**Why:** Frontend code can be faked, replayed, or manipulated. Without real backend validation, a malicious user can do far more than just "test" your app, they can break it.
|
||||
|
||||
3. Never expose your secrets, keep keys truly private (Haven't seen it happening in case of Lovable at least):
|
||||
Accidently exposing env files is common, keeping a tight file security if you're deploying it on your own server.
|
||||
|
||||
4. You can ask your favourite AI vibe-coding tools to generate a security audit tasklist based on your project and follow the tasklist and fix all until finished. That should solve most of the issues.
|
||||
135
Corpus/Various/Setup AnythingLLM with MCP servers.md
Normal file
135
Corpus/Various/Setup AnythingLLM with MCP servers.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
---
|
||||
tags:
|
||||
- llm
|
||||
- mcp
|
||||
- AnythingLLM
|
||||
---
|
||||
|
||||
|
||||
Perplexity, 13 juni 2025
|
||||
|
||||
Prompt: `Can Anything LLM interact with MCP servers? How do I set this up?
|
||||
|
||||
Yes, AnythingLLM can interact with MCP (Model Context Protocol) servers, enabling it to use external tools—including those that work with Obsidian vaults—via standardized APIs. Here’s how you set this up:
|
||||
|
||||
|
||||
|
||||
## How AnythingLLM Connects to MCP Servers
|
||||
|
||||
- **Automatic Detection & Management**: AnythingLLM automatically detects MCP servers and can start, stop, or reload them as needed. You can manage these servers directly from the AnythingLLM UI, specifically through the “Agent Skills” page[^2_1][^2_4].
|
||||
|
||||
- **Configuration File**: The core setup involves editing (or letting AnythingLLM generate) a configuration file named `anythingllm_mcp_servers.json`, found in the `plugins` directory of your AnythingLLM storage[^2_1][^2_4]. This file defines the MCP servers (including command, arguments, environment variables, or HTTP endpoints) that AnythingLLM should connect to.
|
||||
|
||||
- **UI Controls**: Within the UI, you can:
|
||||
|
||||
- Reload/restart MCP servers on the fly
|
||||
- View server status and error logs
|
||||
- Start/stop MCP servers without restarting AnythingLLM[^2_4]
|
||||
|
||||
## Example Configuration
|
||||
|
||||
Here’s a sample snippet for the configuration file:
|
||||
|
||||
```json
|
||||
{"mcpServers":
|
||||
{"obsidian-mcp-server":
|
||||
{ "command": "node",
|
||||
"args": ["path/to/your/obsidian-mcp-server.js"],
|
||||
"env": {"OBSIDIAN_VAULT_PATH": "/path/to/your/vault"}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Or, for an HTTP-based MCP server:
|
||||
|
||||
```json
|
||||
{"mcpServers":
|
||||
{"obsidian-mcp-server":
|
||||
{"url": "http://localhost:8931"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
AnythingLLM will pick up these settings and manage the connection automatically[^2_1][^2_4].
|
||||
## Steps to Set Up
|
||||
|
||||
1. **Install and Configure Your MCP Server**: Set up your Obsidian-compatible MCP server and ensure it’s running (or can be started by AnythingLLM).
|
||||
|
||||
2. **Edit the Configuration File**: Add your MCP server to `anythingllm_mcp_servers.json` as shown above.
|
||||
|
||||
3. **Reload in AnythingLLM**: Go to the “Agent Skills” page in the UI and click “Refresh” to load or reload your MCP servers[^2_4].
|
||||
|
||||
4. **Start/Stop as Needed**: Use the UI controls to start, stop, or debug your MCP server connections without restarting AnythingLLM[^2_4].
|
||||
|
||||
## Additional Notes
|
||||
|
||||
- **Security**: Some MCP servers may require API keys or other authentication, which you can set via environment variables in the config[^2_1].
|
||||
|
||||
- **Multiple Servers**: You can define and manage several MCP servers at once, and AnythingLLM will handle them according to your configuration[^2_1][^2_4].
|
||||
|
||||
- **Extensibility**: AnythingLLM’s MCP support means you can connect to a wide variety of tools, not just those for Obsidian[^2_1][^2_6].
|
||||
|
||||
**Summary:**
|
||||
|
||||
AnythingLLM natively supports MCP servers and provides both configuration file and UI-based management. Just define your desired MCP server in `anythingllm_mcp_servers.json`, reload from the UI, and you’re ready to interact with your Obsidian vault or other tools through standardized MCP APIs[^2_1][^2_4][^2_6].
|
||||
|
||||
|
||||
|
||||
<div style="text-align: center">⁂</div>
|
||||
|
||||
|
||||
|
||||
[^2_1]: https://docs.anythingllm.com/mcp-compatibility/overview
|
||||
|
||||
|
||||
|
||||
[^2_2]: https://apidog.com/blog/beginners-guide-to-mcp-use/
|
||||
|
||||
|
||||
|
||||
[^2_3]: https://dev.to/drunnells/controlling-chrome-with-an-anythingllm-mcp-agent-5891
|
||||
|
||||
|
||||
|
||||
[^2_4]: https://docs.anythingllm.com/mcp-compatibility/desktop
|
||||
|
||||
|
||||
|
||||
[^2_5]: https://github.com/mcp-use/mcp-use
|
||||
|
||||
|
||||
|
||||
[^2_6]: https://github.com/Mintplex-Labs/anything-llm/issues/2883
|
||||
|
||||
|
||||
|
||||
[^2_7]: https://www.reddit.com/r/mcp/comments/1jl8j1n/how_does_an_llm_see_mcp_as_a_client/
|
||||
|
||||
|
||||
|
||||
[^2_8]: https://www.youtube.com/watch?v=QZ6AreMxMKA
|
||||
|
||||
|
||||
|
||||
[^2_9]: https://www.rkatz.xyz/post/2025-04-21-mcp-local-go/
|
||||
|
||||
|
||||
|
||||
[^2_10]: https://github.com/Mintplex-Labs/anything-llm/issues/3000
|
||||
|
||||
|
||||
|
||||
[^2_11]: https://www.youtube.com/watch?v=sMqlObpNz64
|
||||
|
||||
|
||||
|
||||
[^2_12]: https://modelcontextprotocol.io/tutorials/building-mcp-with-llms
|
||||
|
||||
|
||||
|
||||
[^2_13]: https://www.philschmid.de/mcp-example-llama
|
||||
|
||||
|
||||
|
||||
[^2_14]: https://docs.useanything.com
|
||||
10
Corpus/Various/Target Operational Model.md
Normal file
10
Corpus/Various/Target Operational Model.md
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# Elements of a Target Operational Model
|
||||
|
||||
One framework described in the [operating model](https://en.wikipedia.org/wiki/Operating_model "Operating model") definition comes from Ashridge Executive Education – POLISM.[[1]](https://en.wikipedia.org/wiki/Target_operating_model#cite_note-1)[[2]](https://en.wikipedia.org/wiki/Target_operating_model#cite_note-2) This stands for
|
||||
|
||||
- P – processes and capabilities;
|
||||
- O – the organization, i.e. the people that are needed to run the processes or deliver the capabilities, and the organisation structure, accountabilities, incentives and culture that will support and nurture these people;
|
||||
- L – the locations, buildings, infrastructure and other assets and resources needed inside the organisation to support the processes and capabilities;
|
||||
- I – the information systems and other cross-organisation or cross-location links needed to support the processes and capabilities, especially the software applications that are needed to process the information;
|
||||
- S – the suppliers and business partners needed outside the organisation to support the processes and capabilities and the types of agreements between this organisation and these partners.
|
||||
- M – the management systems and processes for developing strategy, planning, setting targets, managing performance and continuous improvement.
|
||||
37
Corpus/Various/The Psychology Behind SaaS Pricing.md
Normal file
37
Corpus/Various/The Psychology Behind SaaS Pricing.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
title: "The Psychology Behind SaaS Pricing That Most Founders Completely Miss"
|
||||
source: "https://www.reddit.com/r/SaaS/comments/1lh0e56/the_psychology_behind_saas_pricing_that_most/?share_id=CYg0pG3jXWObUQLzVP5od&utm_content=2&utm_medium=ios_app&utm_name=iossmf&utm_source=share&utm_term=22"
|
||||
author:
|
||||
- "[[Sea_Reputation_906]]"
|
||||
published: 2025-06-21
|
||||
created: 2025-06-23
|
||||
description:
|
||||
tags:
|
||||
- "clippings"
|
||||
---
|
||||
|
||||
**The anchoring effect: show your expensive plan first**
|
||||
|
||||
When you show your expensive plan first, it makes everything else seem like a bargain. Had a client who was struggling with conversions until we reordered their pricing page to show the premium plan first. Suddenly their middle tier started selling like crazy. People saw the $199/mo plan and thought "well $79 is a steal compared to that!"
|
||||
|
||||
**Freemium gets you freeloaders**
|
||||
|
||||
One client had 10,000+ free users but only like 12 paying customers. Their free plan was way too generous. Another client ditched freemium entirely, switched to a 14-day trial and hit $25K MRR in under 6 months. The difference? People actually had to make a decision instead of sitting in free-user purgatory forever.
|
||||
|
||||
**The $9.99 thing actually works**
|
||||
|
||||
Harvard Business School found that a 1% improvement in pricing can lead to an 11% increase in profit. We've tested this with multiple clients and charm pricing consistently outperforms round numbers.
|
||||
|
||||
**Simpler is always better**
|
||||
|
||||
If your pricing page needs an FAQ section to explain it, you've already lost. Most users won't email to ask questions about your pricing, they'll just bounce. Keep it stupid simple: 2-3 plans max, clear names, bullet points.
|
||||
|
||||
**Higher prices can increase demand (seriously)**
|
||||
|
||||
When you hide your top-tier pricing behind a "contact us" button, it creates weird FOMO for big customers. They imagine they're missing out on some special features. Enterprise leads literally tripled for one client after making this change.
|
||||
|
||||
I see so many founders pricing based on competitors or their costs instead of psychology. The data is clear tho - understanding how people perceive pricing matters way more than your actual costs.
|
||||
|
||||
What pricing experiments have you guys tried? Anything that surprised you?
|
||||
|
||||
(P.S. If you need help building your MVP, DM me. Been getting a ton of requests lately, happy to chat.)
|
||||
118
Corpus/Various/The Ultimate Vibe Coding Guide.md
Normal file
118
Corpus/Various/The Ultimate Vibe Coding Guide.md
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
---
|
||||
title: The Ultimate Vibe Coding Guide
|
||||
source: https://www.reddit.com/r/ClaudeAI/comments/1kivv0w/the_ultimate_vibe_coding_guide/
|
||||
author:
|
||||
- "[[PhraseProfessional54]]"
|
||||
published: 2025-05-10
|
||||
created: 2025-06-10
|
||||
description:
|
||||
tags:
|
||||
- vibecoding
|
||||
- dev
|
||||
- "#type/howto"
|
||||
---
|
||||
|
||||
So I have been using Cursor for more than 6 months now and I find it a very helpful and very strong tool if used correctly and thoughtfully. Through these 6 months and with a lot of fun projects personal and some production-level projects and after more than 2500+ prompts, I learned a lot of tips and tricks that make the development process much easier and faster and makes and help you vibe without so much pain when the codebase gets bigger and I wanted to make a guide for anyone who is new to this and want literally everything in one post and refer to it whenever need any guidance on what to do!:
|
||||
|
||||
# 1\. Define Your Vision Clearly
|
||||
|
||||
**Start with a strong, detailed vision of what you want to build and how it should work.** If your input is vague or messy, the output will be too. Remember: *garbage in, garbage out*. Take time to think through your idea from both a product and user perspective. Use tools like **Gemini 2.5 Pro** in **Google AI Studio** to help structure your thoughts, outline the product goals, and map out how to bring your vision to life. The clearer your plan, the smoother the execution.
|
||||
|
||||
**2\. Plan Your UI/UX First**
|
||||
|
||||
**Before you start building, take time to carefully plan your UI.** Use tools like [v0](https://v0.dev/) to help you visualize and experiment with layouts early. Consistency is key. Decide on your design system upfront and stick with it. Create reusable components such as buttons, loading indicators, and other common UI elements right from the start. This will save you tons of time and effort later on You can also use [\*\*](https://www.google.com/url?sa=E&q=https%3A%2F%2F21st.dev%2F)[https://21st.dev/\*\*](https://21st.dev/**); it has a ton of components with their AI prompts, you just copy-paste the prompt, it is great!
|
||||
|
||||
# 3\. Master Git & GitHub
|
||||
|
||||
**Git is your best friend.** You must know GitHub and Git; it will save you a lot if AI messed things up, you could easily return to an older version. If you did not use Git, your codebase could be destroyed with some wrong changes. You must use it; it makes everything much easier and organized. After finishing a big feature, you must make sure to commit your code. Trust me, this will save you from a lot of disasters in the future!
|
||||
|
||||
# 4\. Choose a Popular Tech Stack
|
||||
|
||||
**Stick to widely-used, well-documented technologies.** AI models are trained on public data. The more common the stack, the better the AI can help you write high-quality code.
|
||||
|
||||
I personally recommend:
|
||||
|
||||
**Next.js** (for frontend and APIs) + **Supabase** (for database and authentication) + **Tailwind CSS** (for styling) + **Vercel** (for hosting).
|
||||
|
||||
This combo is beginner-friendly, fast to develop with, and removes a lot of boilerplate and manual setup.
|
||||
|
||||
# 5\. Utilize Cursor Rules
|
||||
|
||||
**Cursor Rules is your friend.** I am still using it and I think it is still the best solution to start solid. You must have very good Cursor Rules with all the tech stack you are using, instructions to the AI model, best practices, patterns, and some things to avoid. You can find a lot of templates here: [\*\*](https://www.google.com/url?sa=E&q=https%3A%2F%2Fcursor.directory%2F)[https://cursor.directory/\*\*](https://cursor.directory/**)!!
|
||||
|
||||
# 6\. Maintain an Instructions Folder
|
||||
|
||||
**Always have an instructions folder.** It should have markdown files. It should be full of docs-example components to provide to the Ai to guide it better or use (or context7 mcp, it has a tons of documentation).
|
||||
|
||||
# 7\. Craft Detailed Prompts
|
||||
|
||||
Now the building phase starts. You open Cursor and start giving it your prompts. Again, **garbage in, garbage out.** You must give very good prompts. If you cannot, just go plan with Gemini 2.5 Pro on Google AI Studio; make it make a very good intricate version of your prompt. It should be as detailed as possible; do not leave any room for the AI to guess, you must tell it everything.
|
||||
|
||||
# 8\. Break Down Complex Features
|
||||
|
||||
**Do not give huge prompts** like "build me this whole feature." The AI will start to hallucinate and produce shit. You must break down any feature you want to add into phases, especially when you are building a complex feature. Instead of one huge prompt, it should be broken down into 3-5 requests or even more based on your use case.
|
||||
|
||||
# 9\. Manage Chat Context Wisely
|
||||
|
||||
**When the chat gets very big, just open a new one.** Trust me, this is the best. The AI context window is limited; if the chat is very big, it will forget everything earlier, it will forget any patterns, design and will start to produce bad outputs. Just start a new chat window then. When you open the new window, just give the AI a brief description about the feature you were working on and mention the files you were working on. Context is very important (more on that is coming..)!
|
||||
|
||||
# 10\. Don't Hesitate to Restart/Refine Prompts
|
||||
|
||||
When the AI gets it wrong and goes in the wrong way or adding things that you do not want, **returning back, changing the prompt, and sending the AI again would be just much better** than completing on this shit code because AI will try to save its mistakes and will probably introduce new ones. So just return, refine the prompt, and send it again!
|
||||
|
||||
# 11\. Provide Precise Context
|
||||
|
||||
**Providing the right context is the most important thing,** especially when your codebase gets bigger. Mentioning the right files that you know the changes will be made to will save a lot of requests and too much time for you and the AI. But you must make sure these files are relevant because too much context can overwhelm the AI too. You must always make sure to mention the right components that will provide the AI with the context it needs.
|
||||
|
||||
# 12\. Leverage Existing Components for Consistency
|
||||
|
||||
A good trick is that you can **mention previously made components to the AI when building new ones.** The AI will pick up your patterns fast and will use the same in the new component without so much effort!
|
||||
|
||||
# 13\. Iteratively Review Code with AI
|
||||
|
||||
After building each feature, you can take the code of the whole feature, copy-paste it to **Gemini 2.5 Pro** (in Google AI Studio) to check for any security vulnerabilities or bad coding patterns; it has a huge context window. Hence, it actually gives very good insights where you can then input into to **Claude** in Cursor and tell it to fix these flaws. (Tell Gemini to act as a security expert and spot any flaws. In another chat, tell it so you are an expert (in the tech stack at your tech stack), ask it for any performance issues or bad coding patterns). Yeah, it is very good at spotting them! After getting the insights from Gemini, just copy-paste it into Claude to fix any of them, then send it Gemini again until it tells you everything is 100% ok.
|
||||
|
||||
# 14\. Prioritize Security Best Practices
|
||||
|
||||
Regarding security, because it causes a lot of backlash, here are security patterns that you must follow to ensure your website is good and has no very bad security flaws (though it won't be 100% because there will be always flaws in any website by anyone!):
|
||||
|
||||
1. **Trusting Client Data:** Using form/URL input directly.
|
||||
- **Fix:** **Always validate & sanitize on server; escape output.**
|
||||
2. **Secrets in Frontend:** API keys/creds in React/Next.js client code.
|
||||
- **Fix:** **Keep secrets server-side only** (env vars, ensure .env is in .gitignore).
|
||||
3. **Weak Authorization:** Only checking if logged in, not *if allowed* to do/see something.
|
||||
- **Fix:** **Server must verify permissions** for every action & resource.
|
||||
4. **Leaky Errors:** Showing detailed stack traces/DB errors to users.
|
||||
- **Fix:** **Generic error messages for users; detailed logs for devs.**
|
||||
5. **No Ownership Checks (IDOR):** Letting user X access/edit user Y's data via predictable IDs.
|
||||
- **Fix:** **Server must confirm current user owns/can access the specific resource ID.**
|
||||
6. **Ignoring DB-Level Security:** Bypassing database features like RLS for fine-grained access.
|
||||
- **Fix:** **Define data access rules directly in your database** (e.g., RLS).
|
||||
7. **Unprotected APIs & Sensitive Data:** Missing rate limits; sensitive data unencrypted.
|
||||
- **Fix:** **Rate limit APIs (middleware); encrypt sensitive data at rest; always use HTTPS.**
|
||||
|
||||
# 15\. Handle Errors Effectively
|
||||
|
||||
When you face an error, you have two options:
|
||||
|
||||
- Either return back and make the AI do what you asked for again, and yeah this actually works sometimes.
|
||||
- If you want to continue, just copy-paste the error from the console and tell the AI to solve it. But if it took more than three requests without solving it, the best thing to do is returning back again, tweaking your prompt, and providing the correct context as I said before. Correct prompt and right context can save sooo much effort and requests.
|
||||
|
||||
# 16\. Debug Stubborn Errors Systematically
|
||||
|
||||
If there is an error that the AI took so much on and seems never to get it or solve it and started to go on rabbit holes (usually after 3 requests and still did not get it right), **just tell Claude to take an overview of the components the error is coming from and list top suspects it thinks are causing the error.** And also tell it to add logs and then provide the output of them to it again. This will significantly help it find the problem and it works correctly most of the times!
|
||||
|
||||
# 17\. Be Explicit: Prevent Unwanted AI Changes
|
||||
|
||||
Claude has this trait of adding, removing, or modifying things you did not ask for. We all hate it and it sucks. Just a simple sentence under every prompt like **(Do not fuckin change anything I did not ask for Just do only what I fuckin told you)** works very well and it is really effective!
|
||||
|
||||
# 18\. Keep a "Common AI Mistakes" File
|
||||
|
||||
Always have a file of mistakes that you find Claude doing a lot. Add them all to that file and when adding any new feature, just mention that file. This will prevent it from doing any frustrating repeated mistakes and you from repeating yourself!
|
||||
|
||||
I know it does not sound as "vibe coding" anymore and does not sound as easy as all of others describe, but this is actually what you need to do in order to pull off a good project that is useful and usable for a large number of users. These are the most important tips that I learned after using Cursor for more than 6 months and building some projects using it! I hope you found it helpful and if you have any other questions I am happy to help!
|
||||
|
||||
Also, if you made it to here you are a legend and serious about this, so congrats bro!
|
||||
|
||||
Happy vibing!
|
||||
|
||||
BIN
Corpus/Various/Theorie van Gepland Gedrag.png
Normal file
BIN
Corpus/Various/Theorie van Gepland Gedrag.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 165 KiB |
23
Corpus/Various/Theory of planned behavior.md
Normal file
23
Corpus/Various/Theory of planned behavior.md
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
Related: [Fogg Behavior Model](Fogg%20Behavior%20Model.md)
|
||||
|
||||
Elaborated by Icek Ajzen (1985).
|
||||
Wikipedia article [here](https://en.wikipedia.org/wiki/Theory_of_planned_behavior).
|
||||
|
||||
NL artikel op BOOM Strategie [hier](https://boomstrategie.nl/model/gedrag-beinvloeden-met-de-theory-planned-behavior-van-ajzen). (‘Theorie van Gepland Gedrag’)
|
||||
|
||||
De theorie van Ajzen stelt dat bewust gedrag direct voortkomt uit de intentie om het gedrag te vertonen. De intentie wordt volgens Ajzen bepaald door drie elementen:
|
||||
|
||||
_Attitude_
|
||||
|
||||
De attitude gaat over de houding van een persoon ten opzichte van het gedrag. Wanneer de persoon een positieve houding ten opzichte van het gedrag heeft, is de kans groter dat de persoon het gedrag bewust zal vertonen.
|
||||
|
||||
_Subjectieve norm(en)_
|
||||
|
||||
De subjectieve norm gaat over datgene wat de persoon denkt dat anderen – in zijn directe omgeving – vinden van het (uit te voeren) gedrag en hoe zij hierover oordelen. Wanneer de persoon denkt dat anderen het gedrag als normaal of goed beschouwen, is de kans groter dat de persoon het gedrag bewust zal vertonen.
|
||||
|
||||
_Waargenomen gedragscontrole_
|
||||
|
||||
De waargenomen gedragscontrole gaat over de mate waarin de persoon gelooft dat het gedrag eenvoudig uit te voeren is. Dit gaat zowel om de eigen vaardigheden als om de omgevingsfactoren die het gedrag bevorderen of belemmeren. Wanneer de persoon gelooft dat gedrag eenvoudig is uit te voeren, is de kans groter dat de persoon het gedrag bewust zal vertonen.
|
||||
|
||||
**Doorontwikkeling**
|
||||
Het model is niet zonder kritiek. Het model houdt bijvoorbeeld geen rekening met cognitieve en affectieve factoren die van invloed zijn op gedrag. Ajzen heeft daarom samen met sociaal psycholoog Martin Fishbein in 2011 het model doorontwikkeld. Hieruit ontstond de ‘[Reasoned Action Approach](https://www.boomstrategie.nl/reasoned-action-approach-van-ajzen-fishbein)’.
|
||||
8
Corpus/Various/Trias Politica.md
Normal file
8
Corpus/Various/Trias Politica.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
# Trias Politica
|
||||
Bron: [Wikipedia](https://nl.wikipedia.org/wiki/Trias_politica)
|
||||
|
||||
De trias politica, driemachtenleer of scheiding der macht(en) is een theorie van de staatsinrichting waarin de staat opgedeeld is in drie organen die elkaars functioneren bewaken. De gebruikelijke verdeling kent een **wetgevende macht** die wetten opstelt, een **uitvoerende macht** die het dagelijks bestuur van de staat uitoefent volgens de geldige wet en een **rechterlijke macht** die deze uitvoering toetst aan de wet.
|
||||
|
||||
Vgl. met respectievelijk het bestuur/de directie, IT en de compliance officer/FG.
|
||||
Met het verschil dat de CO/FG de uitvoering niet kan blokkeren.
|
||||
|
||||
12
Corpus/Various/Vibe Coding MoC.md
Normal file
12
Corpus/Various/Vibe Coding MoC.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
---
|
||||
tags:
|
||||
- vibecoding
|
||||
- dev
|
||||
- type/MoC
|
||||
---
|
||||
[The Ultimate Vibe Coding Guide](The%20Ultimate%20Vibe%20Coding%20Guide.md)
|
||||
[Vibe Coding Tips](Vibe%20Coding%20Tips.md)
|
||||
[Rules to get better code](Rules%20to%20get%20better%20code.md)
|
||||
[Application Security for Vibe Coding](Application%20Security%20for%20Vibe%20Coding.md)
|
||||
|
||||
https://quick-code-launch.lovable.app/
|
||||
20
Corpus/Various/Vibe Coding Tips.md
Normal file
20
Corpus/Various/Vibe Coding Tips.md
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
tags:
|
||||
- "vibecoding"
|
||||
- "dev"
|
||||
- "prompting"
|
||||
---
|
||||
|
||||
|
||||
1. Write what you're building Before anything, I always start with a small doc called product.md. It says what I’m trying to make, how it should work, and what tools I’m using. Keeps me focused when the AI forgets what I asked.
|
||||
2. Keep notes on how to deploy I got stuck at 1am once trying to remember how I set up my env vars. Now I keep a short file called how-to-ship.txt. Just write it all down early.
|
||||
3. Use git all the time. Push to git early and often. You don’t wanna lose changes when AI goes off script. I push almost every time I finish something. Helps when things break.
|
||||
4. Don’t keep one giant chat Every time I start on a new bug or feature, I open a fresh chat with the AI. It just works better that way. Too much context gets messy.
|
||||
5. Plan features before coding Sometimes I ask the AI to help me think through a flow before I even write code. Then once I get the idea, I start building with smaller prompts.
|
||||
6. Clean your files once a week Delete junk, name stuff better, put things in folders. Blackbox works better when your code is tidy. Also just feels better to look at.
|
||||
7. Don’t ask the AI to build the whole app It’s good with small stuff. UI pieces, simple functions, refactors. Asking it to build your app start to finish usually ends badly.
|
||||
8. Ask questions before asking for code When something breaks, I ask the AI what it thinks first. Let it explain the problem before fixing. Most times it finds the issue faster than me.
|
||||
9. Tech debt comes fast I moved quick with the survey app and the mess built up fast. Take a pause now and then and clean things up or it gets too hard to fix later.
|
||||
10. You’re the one in charge Blackbox is helping but you’re still the one building. Think like a builder. The AI is just there to speed things up when you know what you’re doing. That’s all. Still figuring things out but it’s been fun. If you’re just getting started, hope that helps a bit.
|
||||
11. When building stuff, I often change direction mid-way. I ask ChatGPT to think like a UX designer for ideas, then like a developer, then a security expert. In the end, I always ask how to make the code cleaner and more optimized.
|
||||
12. I realized AI needs a code map to better understand structure—classes, functions, variables, DB schema, relationships, etc. Without it, it starts hallucinating parameters and logic. So I built one for myself and made it public too: [https://codemap4ai.com](https://codemap4ai.com/)
|
||||
14
Corpus/Various/Wat bestuurders willen.md
Normal file
14
Corpus/Various/Wat bestuurders willen.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
---
|
||||
tags:
|
||||
- marketing
|
||||
---
|
||||
|
||||
## Wat bestuurders willen
|
||||
De bestuurders/managers die ik spreek zitten met het volgende:
|
||||
|
||||
- toezichthouders en overheid eisen dat ik mijn beveiliging op orde heb, en er beleid op heb
|
||||
- ik weet niet wat ik moet doen (NIS 2, NEN 7510, BIO)
|
||||
- ik weet niet hoe ik het moet doen en waar ik moet beginnen
|
||||
- ik krijg het niet weggezet in de organisatie, want mijn IT’ers zijn operationeel en hebben geen executiekracht.
|
||||
- ik wil geen technisch verhaal en ik wil geen lange consultancy rapporten
|
||||
- maar ik wil wel weten waar je mee bezig bent, waarom, wat het me gaat kosten en wanneer het af is.
|
||||
196
Corpus/Various/What is an embedding model.md
Normal file
196
Corpus/Various/What is an embedding model.md
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
# What is an embedding model?
|
||||
|
||||
[What are LLM Embeddings? - Iguazio](https://www.iguazio.com/glossary/llm-embeddings/)
|
||||
[Demystifying Embedding Spaces using Large Language Models](https://arxiv.org/html/2310.04475v2)
|
||||
[Demystifying Embedding Spaces using Large Language Models](https://openreview.net/forum?id=qoYogklIPz)
|
||||
[How to Choose the Best Embedding Model for Your LLM Application](https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/)
|
||||
|
||||
An embedding model in the Large Language Model (LLM) space is a neural network component or standalone algorithm that transforms words, phrases, sentences, or even larger pieces of data into dense numerical vectors—called embeddings—that capture the semantic meaning and contextual relationships of the input in a high-dimensional space [1](https://www.iguazio.com/glossary/llm-embeddings/) [5](https://www.couchbase.com/blog/llm-embeddings/) [8](https://aisera.com/blog/llm-embeddings/).
|
||||
|
||||
- Embeddings encode the meaning of text so that semantically similar inputs have vectors that are close together in the embedding space. For example, the words "king" and "queen" will have vectors that are closer to each other than to "apple" [1](https://www.iguazio.com/glossary/llm-embeddings/) [5](https://www.couchbase.com/blog/llm-embeddings/).
|
||||
- Unlike one-hot encoding, which produces sparse and high-dimensional vectors, embeddings are dense and lower-dimensional, making them efficient for computation and storage [1](https://www.iguazio.com/glossary/llm-embeddings/) [6](https://aws.amazon.com/what-is/embeddings-in-machine-learning/).
|
||||
- Modern embedding models, especially those based on transformer architectures like BERT and GPT, generate context-aware embeddings—meaning the representation of a word depends on its surrounding words [5](https://www.couchbase.com/blog/llm-embeddings/) [6](https://aws.amazon.com/what-is/embeddings-in-machine-learning/).
|
||||
- Embeddings are foundational for a wide range of tasks, including search, information retrieval, text classification, recommendation systems, and retrieval-augmented generation (RAG) [1](https://www.iguazio.com/glossary/llm-embeddings/) [4](https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/) [5](https://www.couchbase.com/blog/llm-embeddings/).
|
||||
- The model processes input text through neural network layers (often transformers), mapping each token or sequence into a point in a multi-dimensional vector space [5](https://www.couchbase.com/blog/llm-embeddings/) [6](https://aws.amazon.com/what-is/embeddings-in-machine-learning/).
|
||||
- These vectors are constructed so that the relationships between vectors reflect semantic or contextual similarity.
|
||||
- Embedding models can be trained from scratch or fine-tuned for specific domains (e.g., legal, medical) to improve task performance [5](https://www.couchbase.com/blog/llm-embeddings/).
|
||||
- Finding documents or items similar to a query by comparing embeddings.
|
||||
- Grouping semantically similar items together.
|
||||
- Enhancing LLMs by retrieving relevant context based on embeddings before generating responses [4](https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/) [5](https://www.couchbase.com/blog/llm-embeddings/).
|
||||
- Representing users and items as embeddings to predict preferences [2](https://arxiv.org/html/2310.04475v2).
|
||||
|
||||
| Feature | Description |
|
||||
| --- | --- |
|
||||
| What it produces | Dense, high-dimensional vectors representing input data |
|
||||
| Main advantage | Captures semantic/contextual similarity efficiently |
|
||||
| Typical use cases | Search, retrieval, classification, clustering, recommendations, RAG |
|
||||
| Example models | Word2Vec, BERT, GPT, specialized LLM embedding models |
|
||||
|
||||
In summary, an embedding model in the LLM space is a tool for converting text (or other data) into meaningful vectors that enable efficient, context-aware processing and retrieval across a wide array of AI applications [1](https://www.iguazio.com/glossary/llm-embeddings/) [5](https://www.couchbase.com/blog/llm-embeddings/) [8](https://aisera.com/blog/llm-embeddings/).
|
||||
|
||||
What are the benefits, from a users point of view, of fine-tuning an embedding model on a domain specific text corpus?
|
||||
|
||||
[Improving Retrieval and RAG with Embedding Model Finetuning](https://www.databricks.com/blog/improving-retrieval-and-rag-embedding-model-finetuning)
|
||||
[Fine-Tuning Embedding Models for Domain-Specific Tasks - Modular](https://www.modular.com/ai-resources/fine-tuning-embedding-models-for-domain-specific-tasks
|
||||
[Fine-tuning Embeddings for Domain-Specific NLP - Blog](https://blog.premai.io/fine-tuning-embeddings-for-domain-specific-nlp/)
|
||||
[Fine-tuning Embeddings for Specific Domains - GoPenAI](https://blog.gopenai.com/fine-tuning-embeddings-for-specific-domains-a-comprehensive-guide-5e4298b42185)
|
||||
|
||||
Fine-tuning an embedding model on a domain-specific text corpus offers users tangible improvements in AI-driven applications, particularly in specialized fields like healthcare, law, or finance. Here’s how users benefit:
|
||||
|
||||
## Enhanced Accuracy and Relevance
|
||||
|
||||
- **Domain-specific terminology understanding**: Fine-tuned models better grasp niche vocabulary (e.g., "myocardial infarction" in medicine or "injunction" in law), reducing errors in tasks like document retrieval or question answering [3](https://blog.premai.io/fine-tuning-embeddings-for-domain-specific-nlp/) [4](https://blog.gopenai.com/fine-tuning-embeddings-for-specific-domains-a-comprehensive-guide-5e4298b42185) [5](https://milvus.io/ai-quick-reference/what-is-finetuning-in-embedding-models).
|
||||
- **Improved search results**: Users experience fewer irrelevant results, as embeddings capture contextual nuances (e.g., distinguishing "party" in legal contracts vs. social events) [5](https://milvus.io/ai-quick-reference/what-is-finetuning-in-embedding-models) [8](https://www.thoughtworks.com/radar/techniques/fine-tuning-embedding-models).
|
||||
- **Higher retrieval metrics**: For example, fine-tuned models achieved **~7% performance gains** in retrieval-augmented generation (RAG) systems, directly improving answer quality in applications like customer support or knowledge bases [6](https://www.philschmid.de/fine-tune-embedding-model-for-rag) [1](https://www.databricks.com/blog/improving-retrieval-and-rag-embedding-model-finetuning).
|
||||
|
||||
## Efficiency and Cost Savings
|
||||
|
||||
- **Reduced manual effort**: Automates accurate retrieval of domain-specific content, minimizing time spent sifting through irrelevant data [3](https://blog.premai.io/fine-tuning-embeddings-for-domain-specific-nlp/) [6](https://www.philschmid.de/fine-tune-embedding-model-for-rag).
|
||||
- **Faster training**: Techniques like LoRA (Low-Rank Adaptation) enable efficient fine-tuning on consumer-grade GPUs in minutes, lowering computational costs [3](https://blog.premai.io/fine-tuning-embeddings-for-domain-specific-nlp/) [6](https://www.philschmid.de/fine-tune-embedding-model-for-rag).
|
||||
- **Storage optimization**: Methods like Matryoshka Representation Learning reduce vector storage needs by **6x** while retaining performance [6](https://www.philschmid.de/fine-tune-embedding-model-for-rag).
|
||||
|
||||
## Tailored Solutions for Specialized Needs
|
||||
|
||||
- **Industry-specific performance**: Models adapt to unique requirements, such as legal document analysis or medical diagnosis support, outperforming general-purpose APIs like OpenAI’s in domain benchmarks [1](https://www.databricks.com/blog/improving-retrieval-and-rag-embedding-model-finetuning) [4](https://blog.gopenai.com/fine-tuning-embeddings-for-specific-domains-a-comprehensive-guide-5e4298b42185) [7](https://www.mongodb.com/resources/basics/artificial-intelligence/fine-tuning-embeddings).
|
||||
- **Better alignment with workflows**: Custom embeddings improve downstream tasks like document classification, clustering, and recommendation systems [2](https://www.modular.com/ai-resources/fine-tuning-embedding-models-for-domain-specific-tasks) [5](https://milvus.io/ai-quick-reference/what-is-finetuning-in-embedding-models) [8](https://www.thoughtworks.com/radar/techniques/fine-tuning-embedding-models).
|
||||
|
||||
| **User Benefit** | **Example Use Case** |
|
||||
| --- | --- |
|
||||
| Precise medical QA systems | Retrieving relevant clinical research papers |
|
||||
| Accurate legal contract review | Identifying critical clauses in agreements |
|
||||
| Efficient technical support | Clustering customer tickets by root causes |
|
||||
|
||||
In summary, fine-tuning embedding models translates to **more reliable, efficient, and context-aware AI tools** for users working in specialized domains, directly enhancing productivity and decision-making [1](https://www.databricks.com/blog/improving-retrieval-and-rag-embedding-model-finetuning) [3](https://blog.premai.io/fine-tuning-embeddings-for-domain-specific-nlp/) [5](https://milvus.io/ai-quick-reference/what-is-finetuning-in-embedding-models) [6](https://www.philschmid.de/fine-tune-embedding-model-for-rag).
|
||||
|
||||
## Video on how LLM handles text generation
|
||||
|
||||
[Embeddings 101: The Foundation of LLM Power and Innovation](https://datasciencedojo.com/blog/embeddings-and-llm/)
|
||||
[Understanding LLM Embeddings: A Comprehensive Guide - IrisAgent](https://irisagent.com/blog/understanding-llm-embeddings-a-comprehensive-guide/)
|
||||
Here is a video that clearly explains what happens when you call on a Large Language Model (LLM) for text generation, and how and when an embedding model plays a role:
|
||||
|
||||
- **"How ChatGPT and other LLMs Generate Text?"** (\[YouTube, Super Data Science\] 4)
|
||||
|
||||
This video walks through the step-by-step process of LLM text generation, including:
|
||||
|
||||
- How your input prompt is processed by the model.
|
||||
- The transformation of your text into tokens and then embeddings (vectors) that the neural network can understand.
|
||||
- How the LLM uses these embeddings, along with attention mechanisms, to predict and generate the next most probable word iteratively.
|
||||
- A practical example showing how an LLM answers a question, illustrating the flow from input to generated output.
|
||||
|
||||
The video covers the foundational role of embeddings in converting your input into a machine-readable format, which is then used throughout the model’s architecture to generate coherent and contextually appropriate text 4 [5](https://www.linkedin.com/pulse/how-do-embeddings-work-large-language-model-llm-onfinanceofficial-54awc) [3](https://irisagent.com/blog/understanding-llm-embeddings-a-comprehensive-guide/).
|
||||
|
||||
For a more technical deep dive into embeddings themselves and their role in LLMs, you can also check out the webinar linked in this guide:
|
||||
|
||||
- **"Embeddings 101: The Foundation of LLM Power and Innovation"** (\[Data Science Dojo\] [2](https://datasciencedojo.com/blog/embeddings-and-llm/))
|
||||
|
||||
Both resources will help you visualize and understand the interplay between embedding models and LLMs in text generation.
|
||||
|
||||
how does the embedding model play a role in uploading documents to a workspace in AnythingLLM?[docs.anythingllm.com](https://docs.anythingllm.com/community-hub/upload)
|
||||
|
||||
[
|
||||
|
||||
Uploading to the AnythingLLM Community Hub
|
||||
|
||||
](https://docs.anythingllm.com/community-hub/upload)[docs.useanything](https://docs.useanything.com/llm-not-using-my-docs)
|
||||
|
||||
[
|
||||
|
||||
Why does the LLM not use my documents - AnythingLLM Docs
|
||||
|
||||
](https://docs.useanything.com/llm-not-using-my-docs)
|
||||
|
||||
youtube
|
||||
|
||||
LM Studio + AnythingLLM: Process Local Documents with RAG Like...[Beebom](https://beebom.com/anythingllm-how-chat-with-documents-locally/)
|
||||
|
||||
[
|
||||
|
||||
AnythingLLM Lets You Chat With Documents Locally - Beebom
|
||||
|
||||
](https://beebom.com/anythingllm-how-chat-with-documents-locally/)
|
||||
|
||||
When you upload documents to a workspace in AnythingLLM, the embedding model plays a crucial role in processing and enabling the AI to interact meaningfully with your documents. Here is how it works from a user perspective:
|
||||
|
||||
## Role of the Embedding Model in Document Upload to a Workspace
|
||||
|
||||
1. **Document Processing and Text Extraction**
|
||||
After you upload files (PDFs, TXT, Word, CSV, etc.) to your workspace, AnythingLLM extracts the raw text content from these documents.
|
||||
2. **Chunking the Text**
|
||||
The extracted text is split into smaller, manageable segments or "chunks" to optimize processing and retrieval.
|
||||
3. **Creating Embeddings**
|
||||
Each chunk of text is converted into a dense vector representation using the embedding model. These embeddings capture the semantic meaning and context of the text chunks, enabling the system to understand and compare their content effectively.
|
||||
4. **Storing Embeddings in a Vector Database**
|
||||
The generated embeddings are stored in a vector database linked to your workspace. This database allows fast similarity searches when you query the documents later.
|
||||
5. **Enabling Semantic Search and Chat**
|
||||
When you ask questions or interact with the workspace, the system uses the embeddings to find the most relevant chunks of text by comparing your query’s embedding with those stored from your documents. This retrieval step is often part of retrieval-augmented generation (RAG) workflows. The LLM then generates responses based on the retrieved, semantically relevant content.
|
||||
6. **Updating Embeddings on Document Changes**
|
||||
If you enable features like Automatic Document Sync, the embedding model will re-embed updated documents so that the workspace’s knowledge remains current.
|
||||
|
||||
---
|
||||
|
||||
## Summary of User Workflow with Embeddings in AnythingLLM
|
||||
|
||||
| Step | What Happens with Embeddings |
|
||||
| --- | --- |
|
||||
| Upload document | Text is extracted and chunked |
|
||||
| Process document | Embedding model converts chunks into vectors |
|
||||
| Save to workspace | Vectors stored in vector database for retrieval |
|
||||
| Query workspace | Query converted to embedding; similar vectors found |
|
||||
| Generate response | LLM uses retrieved chunks to produce informed output |
|
||||
|
||||
---
|
||||
|
||||
This embedding process enables AnythingLLM to provide accurate, context-aware answers based on your uploaded documents, supporting efficient local document search and chat functionality without sending data to the cloud [2](https://docs.useanything.com/llm-not-using-my-docs) [4](https://beebom.com/anythingllm-how-chat-with-documents-locally/) [5](https://baresquare.com/blog/keep-your-data-local-using-ollama-and-anythingllm-for-private-ai-interactions) [6](https://ljaweb.com/guides/anythingllm/).
|
||||
|
||||
Would AnythingLLM use the same model for document embedding as output embedding?[docs.anythingllm.com](https://docs.anythingllm.com/setup/embedder-configuration/overview)
|
||||
|
||||
[
|
||||
|
||||
Embedder Configuration - AnythingLLM Docs
|
||||
|
||||
](https://docs.anythingllm.com/setup/embedder-configuration/overview)[docs.useanything.com](https://docs.useanything.com/features/embedding-models)
|
||||
|
||||
[
|
||||
|
||||
Embedding Models - AnythingLLM Docs
|
||||
|
||||
](https://docs.useanything.com/features/embedding-models)[Ajay Ravi's blog](https://ajayravi.hashnode.dev/anything-llm-empowering-conversations-with-your-documents)
|
||||
|
||||
[
|
||||
|
||||
Anything-LLM: Empowering Conversations with Your Documents
|
||||
|
||||
](https://ajayravi.hashnode.dev/anything-llm-empowering-conversations-with-your-documents)[anythingllm](https://anythingllm.com/)
|
||||
|
||||
[
|
||||
|
||||
AnythingLLM | The all-in-one AI application for everyone
|
||||
|
||||
](https://anythingllm.com/)
|
||||
|
||||
In AnythingLLM, **document embeddings** (for processing uploaded files) and **output embeddings** (for generating LLM responses) are handled by **separate systems**, though both rely on embedding models. Here's the breakdown:
|
||||
|
||||
---
|
||||
|
||||
## Document Embedding Model
|
||||
|
||||
- **Single System-wide Model**: AnythingLLM uses one embedding model at a time for **all document processing** (e.g., `all-MiniLM-L6-v2` by default, or alternatives like OpenAI’s `text-embedding-ada-002`) [1](https://docs.anythingllm.com/setup/embedder-configuration/overview) [7](https://gpustack.ai/building-your-private-chatgpt-and-knowledge-base-with-anythingllm-gpustack/).
|
||||
- **Role**: Converts text chunks from uploaded documents into vectors stored in the vector database [6](https://ljaweb.com/guides/anythingllm/).
|
||||
- **Key Constraint**: Changing the embedding model requires re-embedding all documents [1](https://docs.anythingllm.com/setup/embedder-configuration/overview) [2](https://docs.useanything.com/features/embedding-models).
|
||||
|
||||
---
|
||||
|
||||
## Output Embedding (LLM Generation)
|
||||
|
||||
- **Separate LLM Model**: The text generation LLM (e.g., GPT-4, Mistral, or local models) handles **response creation**, which may involve its own internal embeddings for token prediction [5](https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/) [7](https://gpustack.ai/building-your-private-chatgpt-and-knowledge-base-with-anythingllm-gpustack/).
|
||||
- **No Direct Link**: The LLM’s embeddings are distinct from the document embedding model. For example:
|
||||
- You could use `all-MiniLM-L6-v2` for document retrieval while using **Mistral-7B** for response generation [5](https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/) [7](https://gpustack.ai/building-your-private-chatgpt-and-knowledge-base-with-anythingllm-gpustack/).
|
||||
- OpenAI’s LLMs (like GPT-4) don’t share embeddings with their document-focused `text-embedding-ada-002` model [5](https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/) [7](https://gpustack.ai/building-your-private-chatgpt-and-knowledge-base-with-anythingllm-gpustack/).
|
||||
|
||||
---
|
||||
|
||||
## Key Differences
|
||||
|
||||
---
|
||||
|
||||
In summary, AnythingLLM uses **different models** for document embeddings and text generation. The document embedder is system-wide and RAG-focused, while the LLM for responses is independently configurable [1](https://docs.anythingllm.com/setup/embedder-configuration/overview) [5](https://pyimagesearch.com/2024/06/24/integrating-local-llm-frameworks-a-deep-dive-into-lm-studio-and-anythingllm/) [7](https://gpustack.ai/building-your-private-chatgpt-and-knowledge-base-with-anythingllm-gpustack/).
|
||||
29
Corpus/Various/use Warp for coding.md
Normal file
29
Corpus/Various/use Warp for coding.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
You can use Warp for coding by leveraging its built-in AI agents and native code editor, all directly from the terminal. Here’s how you can get started:
|
||||
|
||||
### 1. Start with a Prompt
|
||||
In Warp, you can type a natural language prompt describing what you want to build or change. For example:
|
||||
```
|
||||
Add a function in Python to parse JSON data.
|
||||
```
|
||||
Warp’s agents will interpret your prompt and begin working on the task, generating code, making edits, or suggesting fixes as needed. You can also attach context, such as files or error messages, to guide the agent’s work more precisely [Introducing Warp 2.0](https://www.warp.dev/blog/reimagining-coding-agentic-development-environment).
|
||||
|
||||
### 2. Review and Edit Code Diffs
|
||||
When Warp’s agent generates a code diff, you can review, refine, and apply the changes directly in Warp’s native code editor—no need to switch to an external IDE. The editor supports syntax highlighting for many languages (Python, JavaScript, TypeScript, Rust, etc.) and allows you to save changes with standard shortcuts (`CMD-S` on macOS, `CTRL-S` on Windows/Linux) [Agent Mode Coding Capabilities](https://docs.warp.dev/agents/code).
|
||||
|
||||
### 3. Multitask with Multiple Agents
|
||||
Warp 2.0 allows you to run multiple agents in parallel, so you can work on several coding tasks at once—like building a feature, fixing a bug, and deploying code—all from the same interface [Warp 2.0 Capabilities](https://www.warp.dev/blog/reimagining-coding-agentic-development-environment).
|
||||
|
||||
### 4. Use AI for Suggestions and Corrections
|
||||
You can ask Warp AI for command suggestions, code generation, debugging help, and more. For example, right-click on an error output and select “Ask Warp AI” to get an explanation or fix [Warp AI Overview](https://www.warp.dev/warp-ai).
|
||||
|
||||
### 5. Save and Reuse Workflows
|
||||
With Warp Drive, you can save parameterized commands and workflows for reuse, making it easy to automate repetitive coding tasks or share them with your team [Warp Drive Features](https://www.warp.dev/all-features).
|
||||
|
||||
**Example Coding Tasks in Warp:**
|
||||
- “Write a function in JavaScript to debounce an input.”
|
||||
- “Fix this TypeScript error.”
|
||||
- “Update all instances of ‘var’ to ‘let’ in this file.”
|
||||
- “Add headers to all .py files in this directory”
|
||||
[Agent Mode Coding Examples](https://docs.warp.dev/agents/code)
|
||||
|
||||
Warp is designed to make coding faster and more collaborative, directly from your terminal, with the power of AI agents and a modern editing experience.
|
||||
Loading…
Add table
Add a link
Reference in a new issue