asp.net - The best way to store millions of 100kb/Ave. records -


we in situation need store millions of records everyday,


data structure model:

  • id
  • date
  • title
  • ...
  • data [raw text]

our [raw text] different each time, ~30kb 300kb, , on average it's 100kbs. never need search [raw text], maybe once month data access required of them id.

now storing of them(attributes , data) in mongodb because of great insert speed , performance in mongodb. our database size growing rapidly , it's 85gbs now, , in next few days problem us.

here question, how implement it?
does worth change database , software structures store data[raw text] in file system(/datafiles/x/y/z/id.txt)?
change have significant impact on system performance?

if you're concerned storage, why not compress text data? decent text compression should 10:1.

personally, i'd take file-based approach, because sounds main function archiving. i'd write info file that's needed regenerate database record, compress it, , store in kind of sensible directory structure based on key. reason being it's easy start new disk or move sections of data off archival storage.

if collecting 10 million records each day compression, amounts 100gb per day. might want make 'disk id' form part of key, @ rate you'd fill 2tb disk in 3 weeks. 20tb raid array fill in 6 months.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

c++ - qgraphicsview horizontal scrolling always has a vertical delta -

Image
i've subclassed qgraphicsview custom canvas used in cad application. i've reimplemented qgraphicsview::wheelevent check keyboard modifiers control key and, if control key pressed, zoom. i'm trying implement horizontal scroll when user holds shift , uses wheel. the problem i'm having horizontal scrolling always scrolls 0.279. not huge problem, hugely annoying , points else being wrong. so, here questions: is right way implement horizontal scrolling? if not, is? how eliminate delta of 0.279? thanks in advance. code , sample output below void myview::zoom(int delta) { double factor = pow(1.2, delta/abs(delta)); this->scale(factor, factor); } void myview::scrollhorizontal(int level) { qpointf center = maptoscene(viewport()->rect().center()); qdebug() << "center: " << center.x() << ", " << center.y(); centeron(qpointf(center.x() - level, center.y())); } void myview::wheelevent(qwheeleve...
Read more