第三天:爬蟲【一】


Posted by luckstar77 on 2022-10-27

因為除權息相關的資料需要去抓取網頁資料,因此做了一個爬蟲,步驟如下

  • 設定 vscode debug 環境,在專案根目錄建立一個 .vscode/launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "crawl-stock-dividend-yield",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "program": "${workspaceFolder}/src/index.ts",
            "preLaunchTask": "tsc: build - tsconfig.json",
            "outFiles": ["${workspaceFolder}/build/**/*.js"]
        }
    ]
}
  • 建立 mongodb 連線
import { MongoClient } from 'mongodb';

// Connection URL
const url = 'mongodb://localhost:27017';
const client = new MongoClient(url);

// Database Name
const dbName = 'inventory';

export async function connect() {
    // TODO: error handler

    // Use connect method to connect to the server
    await client.connect();
    console.log('Connected successfully to server');
    const db = client.db(dbName);
    // const collection = db.collection('stock');

    // the following code examples can be pasted here...

    return db;
}
  • 建立 redis 連線
import { createClient } from 'redis';

export async function connect() {

    const client = createClient();

    client.on('error', (err) => console.log('Redis Client Error', err));

    await client.connect();

    return client;
}
  • 將資料庫連線引入到程式流程裡
import { connect as redisConnect } from './db/redis';
import { connect as mongodbConnect } from './db/mongodb';

const COLLECTION = 'stock';

(async () => {
    const mongodbClient = await mongodbConnect();
    const redisClient = await redisConnect();

    const STOCK_ID_INDEX = await redisClient.get('STOCK_ID_INDEX');
    const findResult = await mongodbClient.collection(COLLECTION).find({}).toArray();
})();









Related Posts

Find the Gap - Regression Discontinuity Design

Find the Gap - Regression Discontinuity Design

【JS上課筆記】JavaScript 的非同步請求:AJAX、Fetch API

【JS上課筆記】JavaScript 的非同步請求:AJAX、Fetch API

[ Vue3筆記 ] computed

[ Vue3筆記 ] computed


Comments