第三天:爬蟲【一】


Posted by luckstar77 on 2022-10-27

因為除權息相關的資料需要去抓取網頁資料,因此做了一個爬蟲,步驟如下

  • 設定 vscode debug 環境,在專案根目錄建立一個 .vscode/launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "crawl-stock-dividend-yield",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "program": "${workspaceFolder}/src/index.ts",
            "preLaunchTask": "tsc: build - tsconfig.json",
            "outFiles": ["${workspaceFolder}/build/**/*.js"]
        }
    ]
}
  • 建立 mongodb 連線
import { MongoClient } from 'mongodb';

// Connection URL
const url = 'mongodb://localhost:27017';
const client = new MongoClient(url);

// Database Name
const dbName = 'inventory';

export async function connect() {
    // TODO: error handler

    // Use connect method to connect to the server
    await client.connect();
    console.log('Connected successfully to server');
    const db = client.db(dbName);
    // const collection = db.collection('stock');

    // the following code examples can be pasted here...

    return db;
}
  • 建立 redis 連線
import { createClient } from 'redis';

export async function connect() {

    const client = createClient();

    client.on('error', (err) => console.log('Redis Client Error', err));

    await client.connect();

    return client;
}
  • 將資料庫連線引入到程式流程裡
import { connect as redisConnect } from './db/redis';
import { connect as mongodbConnect } from './db/mongodb';

const COLLECTION = 'stock';

(async () => {
    const mongodbClient = await mongodbConnect();
    const redisClient = await redisConnect();

    const STOCK_ID_INDEX = await redisClient.get('STOCK_ID_INDEX');
    const findResult = await mongodbClient.collection(COLLECTION).find({}).toArray();
})();









Related Posts

【Day01】The Shell(上)

【Day01】The Shell(上)

CSS 常用語法 (Cascading Style Sheets)

CSS 常用語法 (Cascading Style Sheets)

redis 套件的 Property 'on' does not exist on type 'RedisClientType'

redis 套件的 Property 'on' does not exist on type 'RedisClientType'


Comments