Viewing a response to: @apshamilton/re-themarkymark-s8f9c1
Mac Studio and even Mac Minis are very popular option for LLM due to how unified memory works. Nowhere can you get ~188 VRAM for less than the cost of even a single A100 40G.
| author | themarkymark |
|---|---|
| permlink | re-apshamilton-s8f9f0 |
| category | hive-167922 |
| json_metadata | {"tags":["hive-167922"],"app":"peakd/2024.1.1"} |
| created | 2024-02-06 06:45:00 |
| last_update | 2024-02-06 06:45:00 |
| depth | 2 |
| children | 2 |
| last_payout | 2024-02-13 06:45:00 |
| cashout_time | 1969-12-31 23:59:59 |
| total_payout_value | 0.000 HBD |
| curator_payout_value | 0.000 HBD |
| pending_payout_value | 0.000 HBD |
| promoted | 0.000 HBD |
| body_length | 175 |
| author_reputation | 1,780,039,159,558,183 |
| root_title | "Run even larger AI models locally with LM Studio" |
| beneficiaries | [] |
| max_accepted_payout | 1,000,000.000 HBD |
| percent_hbd | 10,000 |
| post_id | 131,072,978 |
| net_rshares | 9,970,475,746 |
| author_curate_reward | "" |
| voter | weight | wgt% | rshares | pct | time |
|---|---|---|---|---|---|
| slider2990 | 0 | 7,404,537,096 | 100% | ||
| astil.codex | 0 | 26,255,061 | 100% | ||
| stemgeeks | 0 | 95,794,074 | 6% | ||
| abh12345.stem | 0 | 1,475,492,633 | 100% | ||
| bilpcoinbot1 | 0 | 124,776,479 | 100% | ||
| yggdrasil.laguna | 0 | 0 | 3% | ||
| saboin.stem | 0 | 843,620,403 | 100% |
I'm getting 23 tokens per second using the 5 bit Mixtal 2.7 model.
| author | apshamilton |
|---|---|
| permlink | re-themarkymark-s8fjax |
| category | hive-167922 |
| json_metadata | {"tags":["hive-167922"],"app":"peakd/2024.1.1"} |
| created | 2024-02-06 10:18:33 |
| last_update | 2024-02-06 10:18:33 |
| depth | 3 |
| children | 1 |
| last_payout | 2024-02-13 10:18:33 |
| cashout_time | 1969-12-31 23:59:59 |
| total_payout_value | 0.034 HBD |
| curator_payout_value | 0.034 HBD |
| pending_payout_value | 0.000 HBD |
| promoted | 0.000 HBD |
| body_length | 66 |
| author_reputation | 212,460,980,305,363 |
| root_title | "Run even larger AI models locally with LM Studio" |
| beneficiaries | [] |
| max_accepted_payout | 1,000,000.000 HBD |
| percent_hbd | 10,000 |
| post_id | 131,076,178 |
| net_rshares | 167,888,678,792 |
| author_curate_reward | "" |
| voter | weight | wgt% | rshares | pct | time |
|---|---|---|---|---|---|
| slider2990 | 0 | 7,255,554,053 | 100% | ||
| amirl | 0 | 158,362,358,831 | 100% | ||
| abh12345.stem | 0 | 1,445,004,935 | 100% | ||
| saboin.stem | 0 | 825,760,973 | 100% |
macs have a big edge for this. I would recommend the 4 bit, the 5 bit isn't much better and takes a lot more ram. I'd stick with 4 bit, or something like 8 bit if you can get there.
| author | themarkymark |
|---|---|
| permlink | re-apshamilton-s8fjcb |
| category | hive-167922 |
| json_metadata | {"tags":["hive-167922"],"app":"peakd/2024.1.1"} |
| created | 2024-02-06 10:19:21 |
| last_update | 2024-02-06 10:19:21 |
| depth | 4 |
| children | 0 |
| last_payout | 2024-02-13 10:19:21 |
| cashout_time | 1969-12-31 23:59:59 |
| total_payout_value | 0.000 HBD |
| curator_payout_value | 0.000 HBD |
| pending_payout_value | 0.000 HBD |
| promoted | 0.000 HBD |
| body_length | 182 |
| author_reputation | 1,780,039,159,558,183 |
| root_title | "Run even larger AI models locally with LM Studio" |
| beneficiaries | [] |
| max_accepted_payout | 1,000,000.000 HBD |
| percent_hbd | 10,000 |
| post_id | 131,076,185 |
| net_rshares | 248,650,609 |
| author_curate_reward | "" |
| voter | weight | wgt% | rshares | pct | time |
|---|---|---|---|---|---|
| astil.codex | 0 | 24,730,557 | 100% | ||
| stemgeeks | 0 | 95,620,147 | 6% | ||
| bilpcoinbot1 | 0 | 128,299,905 | 100% | ||
| yggdrasil.laguna | 0 | 0 | 3% |
hiveblocks